A method and system for continuous learning of fault diagnosis model for substation equipment

Abstract

A method and system for continuous learning of a substation fault diagnosis model belongs to the field of image processing technology and solves the problem of how to enhance the recognition ability of a visual language pre-training model for new and old knowledge. The invention realizes fine-grained image semantic knowledge extraction in a weakly supervised manner and can mine the corresponding relationship of multiple images with similar local features by calculating the global semantic score alignment loss and the local semantic feature contrast loss, so that the model can fully understand and decouple new semantic concepts and learn new knowledge more accurately. By calculating the matching degree between fine-grained text features and fine-grained semantic features, the general knowledge and special knowledge in the new data are decoupled. By decoupling the new semantic concepts, the connection between the new semantic concepts and the existing knowledge is established. Under the premise of maintaining the general knowledge, the special knowledge of the new semantic category is better learned, and the good compatibility of the new and old knowledge is achieved, so that the model has better forward compatibility.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and relates to a method and system for continuous learning of a fault diagnosis model for substation equipment. Background Art

[0002] In recent years, visual language pre-training models represented by the CLIP model have demonstrated strong processing capabilities in visual tasks by pre-training on a large amount of visual text pairing data, and have good zero-sample recognition capabilities for conventional natural image processing. However, for specific downstream tasks, such as image-based substation fault diagnosis tasks, model fine-tuning is required to achieve better recognition results, which will lead to a decrease in the original recognition ability of the model. Therefore, how to achieve continuous improvement in the recognition effect of visual language pre-training models has become the key to model fine-tuning.

[0003] The invention patent with publication number CN118114009A discloses an incremental target detection method based on multimodal data. It uses the strategies of feature space distillation and output space distillation to train the current target detection model, realize the incremental target detection task, and online continuously and incrementally identify and detect the foreground of new categories of targets under the guidance of multimodal information.

[0004] The invention patent with publication number CN118196645A discloses a target recognition method based on multimodal continuous learning. Since the size and number of the target in the image itself are relatively sparse compared to the image background, and the model parameters are also artificially made sparse by analogy, sparse constraints are introduced to avoid catastrophic forgetting problems, improve the multimodal data target detection and recognition capabilities, and reduce the possibility of forgetting old tasks when learning new tasks.

[0005] The invention patent with publication number CN117312909A discloses a task processing method based on continuous learning of prompt word fine-tuning, constructs a preset prompt word pool and prompt word query network model, dynamically assigns the knowledge of the new stage to the prompt word pool, and can perform task processing by using only trace parameters to fine-tune the prompt word to obtain the network model, thereby reducing the required video memory and improving the adaptability and performance of the network model task processing.

[0006] For specific downstream tasks, existing technologies mostly improve the model recognition effect by fixing the original parameters and adding additional low-rank parameters or prompt words to learn new knowledge. However, due to insufficient analysis of the relationship between new and old knowledge, it is easy to confuse new and old knowledge, resulting in conflicts between new and old knowledge. Therefore, how to enhance the recognition ability of visual language pre-training models for new and old knowledge is particularly important in visual tasks. Summary of the invention

[0007] The technical problem to be solved by the present invention is how to enhance the recognition ability of the visual language pre-training model for new and old knowledge.

[0008] The present invention solves the above technical problems through the following technical solutions:

[0009] A method for continuous learning of a fault diagnosis model for a substation equipment comprises the following steps:

[0010] S1. Obtain a training set of substation equipment failures in a single stage, divide it into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0011] S2. Based on the Pytorch deep learning framework, a pre-trained image feature extraction model is established. The model includes a CLIP model and a learnable fine-grained semantic prototype. Fine-grained image semantic knowledge extraction is achieved based on global semantic score alignment and local semantic feature comparison, and the fine-grained semantic prototype is gradient updated.

[0012] S3. Calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation;

[0013] S4, the continuous learning process of substation equipment failures at different stages, continuously updating the pre-trained image feature extraction model, after each stage of training is completed, obtaining the low-rank parameters of the dedicated knowledge of that stage, and obtaining the low-rank parameters used in reasoning through the average fusion of multi-stage low-rank parameters;

[0014] S5. Input the test set and calculate the accuracy of the pre-trained image feature extraction model.

[0015] Further, the S2 includes:

[0016] S21, constructing a CLIP model and a learnable fine-grained semantic prototype; the CLIP model includes a visual branch and text branch , where the visual branch Use ResNet or Transformer structure, text branch The Transformer structure is adopted; the fine-grained semantic prototype , , represents a set of real numbers, where N represents the number of fine-grained semantic prototypes, Represents feature dimension;

[0017] S22, performing appearance enhancement on the original training set images, inputting the original training set images and the enhanced training set images into the visual branch of the CLIP model respectively, calculating the probability scores between the original training set images and the fine-grained semantic prototypes, and the probability scores between the enhanced training set images and the fine-grained semantic prototypes;

[0018] S23, the probability scores between the enhanced training set images and the fine-grained semantic prototypes are obtained through the optimal transmission algorithm to obtain soft pseudo labels, which are used as the supervision signals of the original training set images, and the probability scores between the original training set images and the fine-grained semantic prototypes are obtained through the optimal transmission algorithm to obtain soft pseudo labels, which are used as the supervision signals of the enhanced training set images, and the global semantic score alignment loss is calculated;

[0019] S24. Use fine-grained semantic prototypes to map local image features, calculate query vectors, key vectors, and value vectors, extract local semantic features corresponding to fine-grained semantic features through the attention mechanism, and calculate local semantic feature contrast loss.

[0020] Furthermore, the following logic is used in S22 to respectively input the original training set image and the enhanced training set image into the visual branch of the CLIP model:

[0021]

[0022]

[0023] In the formula, Represents the uth original training set image The image features, Represents the u-th enhanced training set image The image features, Represents the original training set image The enhanced training set images after appearance enhancement are generated by the RandAug library. Represents the visual branch of the CLIP model;

[0024] The probability score between the original training set images and the fine-grained semantic prototypes is calculated using the following logic:

[0025]

[0026] In the formula, express and The cosine similarity between express and The cosine similarity between represents exponential operation, represents the temperature coefficient, represents the probability score between the original training set image and the fine-grained semantic prototype, Representing fine-grained semantic prototypes Middle fine-grained semantic features, Representing fine-grained semantic prototypes Middle Fine-grained semantic features;

[0027] The probability score between the enhanced training set images and the fine-grained semantic prototypes is calculated using the following logic:

[0028]

[0029] In the formula, express and The cosine similarity between express and The cosine similarity between Represents the probability score between the augmented training set images and the fine-grained semantic prototypes.

[0030] Furthermore, the following logic is used in S23 to represent the supervisory signal of the original training set image:

[0031]

[0032] In the formula, represents the supervisory signal of the original training set images, represents the Sinkhorn-Knopp optimal transmission algorithm, represents the probability distribution of the enhanced training set images in the fine-grained semantic prototype, ;

[0033] The following logic is used to enhance the supervisory signal of the training set images:

[0034]

[0035] In the formula, represents the supervisory signal for enhancing the training set images, represents the probability distribution of the original training set images in the fine-grained semantic prototype, ;

[0036] The global semantic score alignment loss is calculated using the following logic:

[0037]

[0038] In the formula, represents the global semantic score alignment loss, represents the KL loss, Indicates the use of measure The difference Indicates the use of measure difference.

[0039] Furthermore, the following logic is used in S24 to query the vector, key vector and value vector:

[0040]

[0041]

[0042]

[0043] In the formula, represents the query vector after mapping, , represents the key vector after mapping, , represents the value vector after mapping, , represents the dimension of the vector after mapping, represents the mth fine-grained semantic feature, , Represents the uth original training set image The local image features, , , represents a learnable mapping matrix;

[0044] The following logic is used to extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism:

[0045]

[0046] In the formula, Representing fine-grained semantic features In the original training set images The local semantic features extracted are represents the transpose of the key vector, Indicates the use of attention mechanism, express function;

[0047] The following logic is used to calculate the local semantic feature contrast loss:

[0048]

[0049] In the formula, represents the local semantic feature contrast loss, cos( )express and The cosine similarity between express and The cosine similarity between Representing fine-grained semantic features In the original training set images The local semantic features extracted are Representing fine-grained semantic features In the The local semantic features extracted from the original training set images, Indicates the size of the training set. Represents the temperature coefficient.

[0050] Further, the S3 includes:

[0051] S31, input the training set labels into Wikipedia or a large language model to obtain fine-grained text descriptions, and input them into the text branch of the CLIP model to obtain fine-grained text features of the category;

[0052] S32, respectively calculating the cosine similarity between all fine-grained text features and each fine-grained semantic feature, if the cosine similarity is greater than 0.5, the corresponding fine-grained semantic feature is regarded as general knowledge, otherwise it is regarded as special knowledge, and a special knowledge image semantic prototype is obtained;

[0053] S33, input the training set image into the visual branch of the CLIP model, obtain the image features of the training set image, calculate the similarity between the image features of the training set image and the semantic prototype of the dedicated knowledge image, obtain the dedicated knowledge image mask, and calculate the image features containing only the dedicated knowledge;

[0054] S34. Input the training set labels into the text branch of the CLIP model to obtain the text features of the training set labels; input the training set images into the visual branch of the CLIP model with low-rank parameters to obtain the image features of the new training set images, and perform a product operation with the special knowledge image mask, and then construct a cross-entropy loss with the text features of the training set labels, and update the low-rank parameters through gradient backpropagation.

[0055] Furthermore, the following logic is used in S31 to obtain the fine-grained text description:

[0056]

[0057] In the formula, Indicates the category The detailed description is A fine-grained text description of represents the cth category, c , G represents the number of categories, represents a detailed description of category e obtained from Wikipedia or a large language model, , represents the number of detailed descriptions retrieved for each artificially set category;

[0058] Use the following logic to obtain the fine-grained text features of this category:

[0059]

[0060] In the formula, Represents fine-grained text description Fine-grained text features.

[0061] Furthermore, the image features of the training set images are obtained using the following logic in S33:

[0062]

[0063] In the formula, represents the training set image of the nth stage, Represents the training set images Image features, x, y represent the indexes of the horizontal and vertical positions of the training set images respectively;

[0064] The following logic is used to calculate the similarity between the image features of the training set images and the semantic prototypes of the dedicated knowledge images:

[0065] sim( )=

[0066] In the formula, sim( ) represents the training set image Image features Similarity with the semantic prototype of the dedicated knowledge image, Represents the semantic features of dedicated knowledge images, , Representing Specialized Knowledge Image Semantic Prototypes , NS represents the number of dedicated image semantic prototypes, Representation calculation and The cosine similarity of

[0067] The dedicated knowledge image mask is obtained using the following logic:

[0068]

[0069] In the formula, Represents a dedicated knowledge image mask;

[0070] The image features containing only specialized knowledge are calculated using the following logic:

[0071]

[0072] In the formula, Represents image features that contain only specialized knowledge.

[0073] Furthermore, the text features of the training set labels are calculated using the following logic in S34:

[0074]

[0075]

[0076] In the formula, Indicates the category The text description of the training set labels, Text features representing the training set labels;

[0077] The following logic is used to calculate the probability that the image feature belongs to the cth category:

[0078]

[0079] In the formula, Represents the training set images The probability of belonging to the cth category, Representation calculation and The cosine similarity of Representation calculation and The cosine similarity of Indicates the category Text features of the training set labels;

[0080] The cross entropy loss is constructed using the following logic:

[0081]

[0082] In the formula, represents the cross entropy loss, Represents the training set images If the true label belongs to the cth category, then ,otherwise 0.

[0083] A continuous learning system for a substation equipment fault diagnosis model, comprising a training set processing module, a knowledge extraction module, a knowledge decoupling module, a continuous learning module, and a test set testing module;

[0084] The training set processing module is used to obtain a training set of substation equipment failures in a single stage, which is divided into a training set and a test set. The training set includes training set images and training set labels, and the test set includes test set images and test set labels. The training set images are randomly cropped, all training set images are scaled to a uniform size, and then numerical normalization is performed.

[0085] The knowledge extraction module is used to establish a pre-trained image feature extraction model based on the Pytorch deep learning framework, wherein the model includes a CLIP model and a learnable fine-grained semantic prototype, implements fine-grained image semantic knowledge extraction based on global semantic score alignment and local semantic feature comparison, and performs gradient update on the fine-grained semantic prototype;

[0086] The knowledge decoupling module is used to calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation;

[0087] The continuous learning module is used for the continuous learning process of substation equipment failures at different stages, and continuously updates the pre-trained image feature extraction model. After the training of each stage is completed, the dedicated knowledge low-rank parameters of the stage are obtained, and the low-rank parameters used in reasoning are obtained by averaging and fusing the low-rank parameters of multiple stages.

[0088] The test set testing module is used to input a test set and calculate the accuracy of the pre-trained image feature extraction model.

[0089] The advantages of the present invention are:

[0090] (1) The present invention realizes fine-grained image semantic knowledge extraction in a weakly supervised manner. By calculating the global semantic score alignment loss and the local semantic feature contrast loss, it can mine the correspondence between multiple images with similar local features, so that the model can fully understand the new semantic concepts and decouple the knowledge, and learn new knowledge more accurately.

[0091] (2) The present invention decouples general knowledge and specialized knowledge in new data by calculating the matching degree between fine-grained text features and fine-grained semantic features. By decoupling new semantic concepts, the concepts are divided into learned general knowledge and unlearned specialized knowledge, and the connection between new semantic concepts and existing knowledge is established. Low-rank parameters are used for learning and fusion, and cross-entropy loss is constructed. Low-rank parameters are updated through gradient backpropagation. Under the premise of maintaining general knowledge, specialized knowledge of new semantic categories can be better learned, and good compatibility of new and old knowledge can be achieved, so that the model has better forward compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is a flow chart of a method for continuous learning of a fault diagnosis model for substation equipment according to a first embodiment of the present invention;

[0093] Figure 2 is a schematic diagram of fine-grained image semantic knowledge extraction according to the first embodiment of the present invention;

[0094] Figure 3 is a schematic diagram of calculating the matching degree between fine-grained text features and fine-grained semantic features according to the first embodiment of the present invention;

[0095] Figure 4 It is a schematic diagram of a continuous learning process of substation faults at different stages according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0097] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0098] Embodiment 1

[0099] like Figure 1 Specifically, a method for continuous learning of a fault diagnosis model for a substation equipment is disclosed, comprising the following steps:

[0100] S1. Obtain a training set of substation equipment failures in a single stage, divide it into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0101] In this embodiment, the continuous learning process is divided into multiple stages. The training set of substation equipment failures will be supplemented with data according to each stage. 80% of the sample data are divided into training sets, and 20% of the sample data are divided into test sets. The training set images are processed separately. The training set of multi-substation equipment defect recognition in a single stage is taken as an example to illustrate. The training set images are randomly cropped, and the size of all training set images is scaled to 256×256, and then numerical normalization is performed.

[0102] S2. Based on the Pytorch deep learning framework, a pre-trained image feature extraction model is established. The model includes a CLIP model and a learnable fine-grained semantic prototype. Fine-grained image semantic knowledge extraction is achieved based on global semantic score alignment and local semantic feature comparison, and the fine-grained semantic prototype is gradient updated.

[0103] like Figure 2 As shown, the S2 further includes the following steps:

[0104] S21, build CLIP model and learnable fine-grained semantic prototype;

[0105] In this embodiment, the CLIP model includes a visual branch and text branch , where the visual branch Use ResNet or Transformer structure, text branch Adopt Transformer structure;

[0106] In this embodiment, the dimension of the image features of the training set images is , building learnable fine-grained semantic prototypes , represents the set of real numbers, ,in, Represents the first fine-grained semantic features, , N represents the number of fine-grained semantic prototypes, Indicates feature dimensions. In this embodiment, fine-grained semantic features is randomly initialized and updated by interacting with image features.

[0107] S22, performing appearance enhancement on the original training set images, inputting the original training set images and the enhanced training set images into the visual branch of the CLIP model respectively, calculating the probability scores between the original training set images and the fine-grained semantic prototypes, and the probability scores between the enhanced training set images and the fine-grained semantic prototypes;

[0108] In this embodiment, the original training set images , the enhanced training set images after appearance enhancement are generated by RandAug library , respectively, the original training set images And the enhanced training set images Input the visual branch of the CLIP model to obtain the original training set image And the enhanced training set images image features.

[0109] In this embodiment, the following logic is used to respectively input the original training set images and the enhanced training set images into the visual branch of the CLIP model:

[0110]

[0111]

[0112] In the formula, Represents the uth original training set image The image features, Represents the u-th enhanced training set image image features.

[0113] In this embodiment, the probability score between the original training set image and the fine-grained semantic prototype is calculated using the following logic:

[0114]

[0115] In the formula, express and The cosine similarity between express and The cosine similarity between represents exponential operation, represents the temperature coefficient, Represents the probability score between the original training set image and the fine-grained semantic prototype, multiple The complete probability distribution can be obtained .

[0116] The probability score between the enhanced training set images and the fine-grained semantic prototypes is calculated using the following logic:

[0117]

[0118] In the formula, express and The cosine similarity between express and The cosine similarity between Represents the probability score between the enhanced training set image and the fine-grained semantic prototype, multiple The complete probability distribution can be obtained .

[0119] S23, the probability scores between the enhanced training set images and the fine-grained semantic prototypes are obtained through the optimal transmission algorithm to obtain soft pseudo labels, which are used as the supervision signals of the original training set images, and the probability scores between the original training set images and the fine-grained semantic prototypes are obtained through the optimal transmission algorithm to obtain soft pseudo labels, which are used as the supervision signals of the enhanced training set images, and the global semantic score alignment loss is calculated;

[0120] In this embodiment, in order to ensure the accuracy of the model recognition effect, the probability scores between the enhanced training set images and the original training set images and the fine-grained semantic prototypes should be consistent. This embodiment adopts the Sinkhorn-Knopp optimal transmission algorithm to obtain a soft pseudo label for one view of the enhanced training set images and the original training set images, respectively, as a supervisory signal for the other view, so that all images should be evenly assigned to different fine-grained semantic features.

[0121] In this embodiment, the following logic is used to represent the supervisory signal of the original training set image:

[0122]

[0123] In the formula, represents the supervisory signal of the original training set images, represents the Sinkhorn-Knopp optimal transmission algorithm, Represents the probability distribution of the enhanced training set images in fine-grained semantic prototypes.

[0124] In this embodiment, the following logic is used to enhance the supervisory signal of the training set image:

[0125]

[0126] In the formula, represents the supervisory signal for enhancing the training set images, Represents the probability distribution of the original training set images in fine-grained semantic prototypes.

[0127] In this embodiment, the global semantic score alignment loss is calculated using the following logic:

[0128]

[0129] In the formula, represents the global semantic score alignment loss, represents the KL loss, Indicates the use of measure The difference Indicates the use of measure difference.

[0130] S24, mapping local image features using fine-grained semantic prototypes, calculating query vectors, key vectors, and value vectors, extracting local semantic features corresponding to the fine-grained semantic features through an attention mechanism, and calculating local semantic feature contrast loss;

[0131] In this embodiment, the fine-grained semantic prototype is used as the query, and the u-th original training set image Local image features As key and value, ,in, Represents the uth original training set image Image features of some areas, L represents the number of local semantic features, d represents the dimension of the features, and the local image features are mapped using fine-grained semantic prototypes to calculate the query vector , key vector Sum value vector , extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism, and calculate the contrast loss of the local semantic features.

[0132] In this embodiment, the following logic query vector is used: , key vector Sum value vector :

[0133]

[0134]

[0135]

[0136] In the formula, represents the query vector after mapping, , represents the key vector after mapping, , represents the value vector after mapping, , represents the dimension of the vector after mapping, represents the mth fine-grained semantic feature, , Represents the uth original training set image The local image features, , , represents the learnable mapping matrix.

[0137] In this embodiment, the following logic is used to extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism:

[0138]

[0139] In the formula, Representing fine-grained semantic features In the original training set images The local semantic features extracted are represents the transpose of the key vector, Indicates the use of attention mechanism, express function.

[0140] For the uth original training set image and the sth original training set image , and the corresponding labels are , ,like ,represent and are images of the same category, and The responses of the same fine-grained semantic feature are similar, and the responses of images of other categories to the fine-grained semantic feature are far away. Therefore, by calculating the local semantic feature contrast loss, multiple images of the same category can be mined.

[0141] In this embodiment, the following logic is used to calculate the local semantic feature contrast loss:

[0142]

[0143] In the formula, represents the local semantic feature contrast loss, cos( )express and The cosine similarity between express and The cosine similarity between Representing fine-grained semantic features In the original training set images The local semantic features extracted are Representing fine-grained semantic features In the The local semantic features extracted from the original training set images, Indicates the size of the training set. Indicates the temperature coefficient. In this embodiment, 0.07.

[0144] In this embodiment, the image features As an image The global features represent the features after the spatial information of the entire image is averaged, while the local semantic features Represents the features of part of the image area, fine-grained semantic features In fact, corresponding to specific local features, for example, oil level gauges of transformers of different models may have similar local features.

[0145] Fine-grained semantic knowledge helps the model to fully understand new semantic concepts and decouple knowledge. Fine-grained descriptions of text features can be obtained in a variety of ways, while visual features need to take into account the diversity and complexity of image data. This embodiment extracts fine-grained semantic knowledge of images in a weakly supervised manner. By calculating the global semantic score alignment loss and the local semantic feature contrast loss, it can mine the correspondence between multiple images with similar local features, and assist in knowledge decoupling based on fine-grained feature matching of image and text.

[0146] S3. Calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation;

[0147] like Figure 3 As shown, the S3 further includes the following steps:

[0148] S31, input the training set labels into Wikipedia or a large language model to obtain fine-grained text descriptions, and input them into the text branch of the CLIP model to obtain fine-grained text features of the category;

[0149] In this embodiment, a detailed text description of the training set label (new semantics) is obtained with the help of tools such as a large language model and Wikipedia, and the text description is processed through the text branch of the CLIP model to obtain fine-grained text features of the category.

[0150] In this embodiment, the following logic is used to obtain the fine-grained text description:

[0151]

[0152] In the formula, Indicates the category The detailed description is A fine-grained text description of represents the cth category, represents a detailed description of category e obtained from Wikipedia or a large language model, , represents the number of detailed descriptions retrieved for each category set manually. In this embodiment, 10.

[0153] In this embodiment, the following logic is used to obtain the fine-grained text features of this category:

[0154]

[0155] In the formula, Represents fine-grained text description Fine-grained text features.

[0156] S32, respectively calculating the cosine similarity between all fine-grained text features and each fine-grained semantic feature, if the cosine similarity is greater than 0.5, the corresponding fine-grained semantic feature is regarded as general knowledge, otherwise it is regarded as special knowledge, and a special knowledge image semantic prototype is obtained;

[0157] In this embodiment, all fine-grained text features are calculated separately With each fine-grained semantic feature The cosine similarity of ,in, , N represents the number of fine-grained semantic prototypes, c , G represents the number of categories, , Indicates the number of detailed descriptions retrieved for each category set by humans; if any and Satisfy cosine similarity 0.5, the corresponding fine-grained semantic features As general knowledge, otherwise as special knowledge, thus obtaining the special knowledge image semantic prototype , where NS represents the number of dedicated image semantic prototypes.

[0158] S33, input the training set image into the visual branch of the CLIP model, obtain the image features of the training set image, calculate the similarity between the image features of the training set image and the semantic prototype of the dedicated knowledge image, obtain the dedicated knowledge image mask, and calculate the image features containing only the dedicated knowledge;

[0159] In this embodiment, the image features of the training set images are obtained using the following logic:

[0160]

[0161] In the formula, represents the training set image of the nth stage, Represents the training set images image features, x and y represent the indexes of the horizontal and vertical positions of the training set images respectively.

[0162] In this embodiment, the following logic is used to calculate the similarity between the image features of the training set images and the semantic prototypes of the special knowledge images:

[0163] sim( )=

[0164] In the formula, sim( ) represents the training set image Image features Similarity with the semantic prototype of the dedicated knowledge image, Represents the semantic features of dedicated knowledge images, , Representation calculation and The cosine similarity of .

[0165] In this embodiment, the dedicated knowledge image mask is obtained using the following logic:

[0166]

[0167] In the formula, Represents a specialized knowledge image mask.

[0168] In this embodiment, the following logic is used to calculate image features that only contain specialized knowledge:

[0169]

[0170] In the formula, Represents image features that contain only specialized knowledge.

[0171] S34, input the training set labels into the text branch of the CLIP model to obtain the text features of the training set labels; input the training set images into the visual branch of the CLIP model with low-rank parameters to obtain the image features of the new training set images, and perform product operations with the dedicated knowledge image mask, and then construct a cross entropy loss with the text features of the training set labels, and update the low-rank parameters through gradient backpropagation;

[0172] In this embodiment, the text features of the training set labels are calculated using the following logic:

[0173]

[0174]

[0175] In the formula, Indicates the category The text description of the training set labels, Text features representing the training set labels.

[0176] In this embodiment, low-rank (LoRA) parameters are used to learn the model at each stage of continuous learning, and the probability that the image feature belongs to the cth category is calculated using the following logic:

[0177]

[0178] In the formula, Represents the training set images The probability of belonging to the cth category, Representation calculation and The cosine similarity of Representation calculation and The cosine similarity of Indicates the category The text features of the training set labels.

[0179] In this embodiment, a cross entropy loss is constructed based on the above probability to guide the learning of low-rank parameters, so that the low-rank parameters of each stage can learn specific semantic knowledge. The cross entropy loss is constructed using the following logic:

[0180]

[0181] In the formula, represents the cross entropy loss, Represents the training set images If the true label belongs to the cth category, then ,otherwise 0.

[0182] This embodiment performs similarity matching between fine-grained text features and fine-grained semantic features of images. The degree of matching represents whether the pre-trained image feature extraction model contains the corresponding knowledge, thereby screening out special knowledge image semantic prototypes, realizing knowledge decoupling of general knowledge and special knowledge, and establishing corresponding low-rank parameters in the visual branch of the model for learning and fusion.

[0183] S4. Continuously learn about substation equipment failures at different stages and continuously update the pre-trained image feature extraction model. After each stage of training is completed, obtain the low-rank parameters of the dedicated knowledge of that stage. Through the average fusion of multi-stage low-rank parameters, obtain the low-rank parameters used in reasoning.

[0184] In this embodiment, steps S1 to S3 only consider the substation fault image data of a single stage in the continuous learning process, and the continuous learning process of substation faults at different stages is the same. Based on the continuous learning process at different stages, the pre-trained image feature extraction model is continuously updated. Figure 4As shown in the figure, after each stage of training is completed, the dedicated knowledge low-rank parameters of that stage will be obtained. Different dedicated knowledge low-rank parameters preserve the knowledge of the data in that stage and do not interfere with each other. Through the average fusion of multi-stage low-rank parameters, the dedicated knowledge low-rank parameters of different stages are averaged to obtain the low-rank parameters used in reasoning, and then a visual language model with continuously improved semantic generalization ability is obtained.

[0185] S5. Input the test set and calculate the accuracy of the pre-trained image feature extraction model.

[0186] In this embodiment, the pre-trained image feature extraction model is trained in multiple stages, test set images and test set labels are input, and the accuracy of the pre-trained image feature extraction model is calculated.

[0187] Embodiment 2

[0188] The present invention also provides a continuous learning system for a substation equipment fault diagnosis model, comprising a training set processing module, a knowledge extraction module, a knowledge decoupling module, a continuous learning module, and a test set testing module;

[0189] The training set processing module is used to obtain a training set of substation equipment failures in a single stage, which is divided into a training set and a test set. The training set includes training set images and training set labels, and the test set includes test set images and test set labels. The training set images are randomly cropped, all training set images are scaled to a uniform size, and then numerical normalization is performed.

[0190] In this embodiment, the continuous learning process is divided into multiple stages. The training set of substation equipment failure will be supplemented with data according to each stage. The training set of substation equipment failure in a single stage is taken as an example for explanation. The training set includes images and corresponding labels. The training set images are randomly cropped in a conventional way, and the size of all training set images is scaled to 256×256, and then numerical normalization is performed.

[0191] The knowledge extraction module is used to establish a pre-trained image feature extraction model based on the Pytorch deep learning framework, wherein the model includes a CLIP model and a learnable fine-grained semantic prototype, implements fine-grained image semantic knowledge extraction based on global semantic score alignment and local semantic feature comparison, and performs gradient update on the fine-grained semantic prototype;

[0192] The knowledge extraction module also includes the following units:

[0193] Modeling unit, used to build CLIP models and learnable fine-grained semantic prototypes;

[0194] In this embodiment, the CLIP model includes a visual branch and text branch , where the visual branch Use ResNet or Transformer structure, text branch Adopt Transformer structure;

[0195] In this embodiment, the dimension of the image features of the training set images is , building learnable fine-grained semantic prototypes , represents the set of real numbers, ,in, Represents the first fine-grained semantic features, , N represents the number of fine-grained semantic prototypes, Indicates feature dimensions. In this embodiment, fine-grained semantic features is randomly initialized and updated by interacting with image features.

[0196] The probability score unit is used to enhance the appearance of the original training set images. The original training set images and the enhanced training set images are respectively input into the visual branch of the CLIP model to calculate the probability scores between the original training set images and the fine-grained semantic prototypes, and the probability scores between the enhanced training set images and the fine-grained semantic prototypes.

[0197] In this embodiment, the original training set images , the enhanced training set images after appearance enhancement are generated by RandAug library , respectively, the original training set images And the enhanced training set images Input the visual branch of the CLIP model to obtain the original training set image And the enhanced training set images image features.

[0198] In this embodiment, the following logic is used to respectively input the original training set images and the enhanced training set images into the visual branch of the CLIP model:

[0199]

[0200]

[0201] In the formula, Represents the uth original training set image The image features, Represents the u-th enhanced training set image image features.

[0202] In this embodiment, the probability score between the original training set image and the fine-grained semantic prototype is calculated using the following logic:

[0203]

[0204] In the formula, express and The cosine similarity between express and The cosine similarity between represents exponential operation, represents the temperature coefficient, Represents the probability score between the original training set image and the fine-grained semantic prototype, multiple The complete probability distribution can be obtained .

[0205] The probability score between the enhanced training set images and the fine-grained semantic prototypes is calculated using the following logic:

[0206]

[0207] In the formula, express and The cosine similarity between express and The cosine similarity between Represents the probability score between the enhanced training set image and the fine-grained semantic prototype, multiple The complete probability distribution can be obtained .

[0208] The global unit is used to obtain soft pseudo labels by using the optimal transmission algorithm for the probability scores between the enhanced training set images and the fine-grained semantic prototypes, which are used as the supervision signals for the original training set images. The global unit is used to obtain soft pseudo labels by using the optimal transmission algorithm for the probability scores between the original training set images and the fine-grained semantic prototypes, which are used as the supervision signals for the enhanced training set images, and calculate the global semantic score alignment loss.

[0209] In this embodiment, in order to ensure the accuracy of the model recognition effect, the probability scores between the enhanced training set images and the original training set images and the fine-grained semantic prototypes should be consistent. This embodiment adopts the Sinkhorn-Knopp optimal transmission algorithm to obtain a soft pseudo label for one view of the enhanced training set images and the original training set images, respectively, as a supervisory signal for the other view, so that all images should be evenly assigned to different fine-grained semantic features.

[0210] In this embodiment, the following logic is used to represent the supervisory signal of the original training set image:

[0211]

[0212] In the formula, represents the supervisory signal of the original training set images, represents the Sinkhorn-Knopp optimal transmission algorithm, Represents the probability distribution of the enhanced training set images in fine-grained semantic prototypes.

[0213] In this embodiment, the following logic is used to enhance the supervisory signal of the training set image:

[0214]

[0215] In the formula, represents the supervisory signal for enhancing the training set images, Represents the probability distribution of the original training set images in fine-grained semantic prototypes.

[0216] In this embodiment, the global semantic score alignment loss is calculated using the following logic:

[0217]

[0218] In the formula, represents the global semantic score alignment loss, represents the KL loss, Indicates the use of measure The difference Indicates the use of measure difference.

[0219] The local unit is used to map local image features using fine-grained semantic prototypes, calculate query vectors, key vectors, and value vectors, extract local semantic features corresponding to fine-grained semantic features through the attention mechanism, and calculate the local semantic feature contrast loss;

[0220] In this embodiment, the fine-grained semantic prototype is used as the query, and the u-th original training set image Local image features As key and value, ,in, Represents the uth original training set image Image features of some areas, L represents the number of local semantic features, d represents the dimension of the features, and the local image features are mapped using fine-grained semantic prototypes to calculate the query vector , key vector Sum value vector , extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism, and calculate the contrast loss of the local semantic features.

[0221] In this embodiment, the following logic query vector is used: , key vector Sum value vector :

[0222]

[0223]

[0224]

[0225] In the formula, represents the query vector after mapping, , represents the key vector after mapping, , represents the value vector after mapping, , represents the dimension of the vector after mapping, represents the mth fine-grained semantic feature, , Represents the uth original training set image The local image features, , , represents the learnable mapping matrix.

[0226] In this embodiment, the following logic is used to extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism:

[0227]

[0228] In the formula, Representing fine-grained semantic features In the original training set images The local semantic features extracted are represents the transpose of the key vector, Indicates the use of attention mechanism, express function.

[0229] For the uth original training set image and the sth original training set image , and the corresponding labels are , ,like ,represent and are images of the same category, and The responses of the same fine-grained semantic feature are similar, and the responses of images of other categories to the fine-grained semantic feature are far away. Therefore, by calculating the local semantic feature contrast loss, multiple images of the same category can be mined.

[0230] In this embodiment, the following logic is used to calculate the local semantic feature contrast loss:

[0231]

[0232] In the formula, represents the local semantic feature contrast loss, cos( )express and The cosine similarity between express and The cosine similarity between Representing fine-grained semantic features In the original training set images The local semantic features extracted are Representing fine-grained semantic features In the The local semantic features extracted from the original training set images, Indicates the size of the training set. Indicates the temperature coefficient. In this embodiment, 0.07.

[0233] In this embodiment, the image features As an image The global features represent the features after the spatial information of the entire image is averaged, while the local semantic features Represents the features of part of the image area, fine-grained semantic features In fact, corresponding to specific local features, for example, oil level gauges of transformers of different models may have similar local features.

[0234] Fine-grained semantic knowledge helps the model to fully understand new semantic concepts and decouple knowledge. Fine-grained descriptions of text features can be obtained in a variety of ways, while visual features need to take into account the diversity and complexity of image data. This embodiment extracts fine-grained semantic knowledge of images in a weakly supervised manner. By calculating the global semantic score alignment loss and the local semantic feature contrast loss, it can mine the correspondence between multiple images with similar local features, and assist in knowledge decoupling based on fine-grained feature matching of image and text.

[0235] The knowledge decoupling module is used to calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation;

[0236] The knowledge decoupling module also includes the following units:

[0237] The text feature unit is used to input the training set labels into Wikipedia or a large language model to obtain fine-grained text descriptions, and then input them into the text branch of the CLIP model to obtain fine-grained text features of the category.

[0238] In this embodiment, a detailed text description of the training set label (new semantics) is obtained with the help of tools such as a large language model and Wikipedia, and the text description is processed through the text branch of the CLIP model to obtain fine-grained text features of the category.

[0239] In this embodiment, the following logic is used to obtain the fine-grained text description:

[0240]

[0241] In the formula, Indicates the category The detailed description is A fine-grained text description of represents the cth category, represents a detailed description of category e obtained from Wikipedia or a large language model, , represents the number of detailed descriptions retrieved for each category set manually. In this embodiment, 10.

[0242] In this embodiment, the following logic is used to obtain the fine-grained text features of this category:

[0243]

[0244] In the formula, Represents fine-grained text description Fine-grained text features.

[0245] The dedicated knowledge unit is used to calculate the cosine similarity between all fine-grained text features and each fine-grained semantic feature. If the cosine similarity is greater than 0.5, the corresponding fine-grained semantic feature is used as general knowledge, otherwise it is used as dedicated knowledge to obtain a dedicated knowledge image semantic prototype.

[0246] In this embodiment, all fine-grained text features are calculated separately With each fine-grained semantic feature The cosine similarity of ,in, , N represents the number of fine-grained semantic prototypes, c , G represents the number of categories, , Indicates the number of detailed descriptions retrieved for each category set by humans; if any and Satisfy cosine similarity 0.5, the corresponding fine-grained semantic features As general knowledge, otherwise as special knowledge, thus obtaining the special knowledge image semantic prototype , where NS represents the number of dedicated image semantic prototypes.

[0247] The image feature unit inputs the training set images into the visual branch of the CLIP model, obtains the image features of the training set images, calculates the similarity between the image features of the training set images and the semantic prototype of the dedicated knowledge image, obtains the dedicated knowledge image mask, and calculates the image features that only contain the dedicated knowledge;

[0248] In this embodiment, the image features of the training set images are obtained using the following logic:

[0249]

[0250] In the formula, represents the training set image of the nth stage, Represents the training set images image features, x and y represent the indexes of the horizontal and vertical positions of the training set images respectively.

[0251] In this embodiment, the following logic is used to calculate the similarity between the image features of the training set images and the semantic prototypes of the special knowledge images:

[0252] sim( )=

[0253] In the formula, sim( ) represents the training set image Image features Similarity with the semantic prototype of the dedicated knowledge image, Represents the semantic features of dedicated knowledge images, , Representation calculation and The cosine similarity of .

[0254] In this embodiment, the dedicated knowledge image mask is obtained using the following logic:

[0255]

[0256] In the formula, Represents a specialized knowledge image mask.

[0257] In this embodiment, the following logic is used to calculate image features that only contain specialized knowledge:

[0258]

[0259] In the formula, Represents image features that contain only specialized knowledge.

[0260] The cross entropy loss unit is used to input the training set labels into the text branch of the CLIP model to obtain the text features of the training set labels; the training set images are input into the visual branch of the CLIP model with low-rank parameters to obtain the image features of the new training set images, and the image features are multiplied with the special knowledge image mask, and then the cross entropy loss is constructed with the text features of the training set labels, and the low-rank parameters are updated through gradient backpropagation;

[0261] In this embodiment, the text features of the training set labels are calculated using the following logic:

[0262]

[0263]

[0264] In the formula, Indicates the category The text description of the training set labels, Text features representing the training set labels.

[0265] In this embodiment, low-rank (LoRA) parameters are used to learn the model at each stage of continuous learning, and the probability that the image feature belongs to the cth category is calculated using the following logic:

[0266]

[0267] In the formula, Represents the training set images The probability of belonging to the cth category, Representation calculation and The cosine similarity of Representation calculation and The cosine similarity of Indicates the category The text features of the training set labels.

[0268] In this embodiment, a cross entropy loss is constructed based on the above probability to guide the learning of low-rank parameters, so that the low-rank parameters of each stage can learn specific semantic knowledge. The cross entropy loss is constructed using the following logic:

[0269]

[0270] In the formula, represents the cross entropy loss, Represents the training set images If the true label belongs to the cth category, then ,otherwise 0.

[0271] This embodiment performs similarity matching between fine-grained text features and fine-grained semantic features of images. The degree of matching represents whether the pre-trained image feature extraction model contains the corresponding knowledge, thereby screening out special knowledge image semantic prototypes, realizing knowledge decoupling of general knowledge and special knowledge, and establishing corresponding low-rank parameters in the visual branch of the model for learning and fusion.

[0272] The continuous learning module is used for the continuous learning process of substation equipment failures at different stages, continuously updating the pre-trained image feature extraction model, and obtaining the dedicated knowledge low-rank parameters of each stage after the training is completed. The low-rank parameters used for reasoning are obtained by averaging and fusing the multi-stage low-rank parameters.

[0273] In this embodiment, steps S1 to S3 only consider the substation fault image data of a single stage in the continuous learning process, and the continuous learning process of substation faults at different stages is the same. Based on the continuous learning process at different stages, the pre-trained image feature extraction model is continuously updated. Figure 4 As shown in the figure, after each stage of training is completed, the dedicated knowledge low-rank parameters of that stage will be obtained. Different dedicated knowledge low-rank parameters preserve the knowledge of the data in that stage and do not interfere with each other. Through the average fusion of multi-stage low-rank parameters, the dedicated knowledge low-rank parameters of different stages are averaged to obtain the low-rank parameters used in reasoning, and then a visual language model with continuously improved semantic generalization ability is obtained.

[0274] The test set testing module is used to input a test set and calculate the accuracy of the pre-trained image feature extraction model.

[0275] In this embodiment, the pre-trained image feature extraction model is trained in multiple stages, test set images and test set labels are input, and the accuracy of the pre-trained image feature extraction model is calculated.

[0276] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for continuous learning of a fault diagnosis model for substation equipment, characterized in that: include: S1. Obtain a training set of substation equipment failures in a single stage, divide it into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing; S2. Based on the Pytorch deep learning framework, a pre-trained image feature extraction model is established. The model includes a CLIP model and a learnable fine-grained semantic prototype. Fine-grained image semantic knowledge extraction is achieved based on global semantic score alignment and local semantic feature comparison, and the fine-grained semantic prototype is gradient updated. The S2 includes: S21, constructing a CLIP model and a learnable fine-grained semantic prototype; the CLIP model includes a visual branch and text branch ; S22, performing appearance enhancement on the original training set images, inputting the original training set images and the enhanced training set images into the visual branch of the CLIP model respectively, calculating the probability scores between the original training set images and the fine-grained semantic prototypes, and the probability scores between the enhanced training set images and the fine-grained semantic prototypes; S23, the probability scores between the enhanced training set images and the fine-grained semantic prototypes are transferred through the optimal transmission algorithm to obtain the supervision signal of the original training set images, the probability scores between the original training set images and the fine-grained semantic prototypes are transferred through the optimal transmission algorithm to obtain the supervision signal of the enhanced training set images, and the global semantic score alignment loss is calculated; S24, mapping local image features using fine-grained semantic prototypes, calculating query vectors, key vectors, and value vectors, extracting local semantic features corresponding to the fine-grained semantic features through an attention mechanism, and calculating local semantic feature contrast loss; S3. Calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation; S4, the continuous learning process of substation equipment failures at different stages, continuously updating the pre-trained image feature extraction model, after each stage of training is completed, obtaining the low-rank parameters of the dedicated knowledge of that stage, and obtaining the low-rank parameters used in reasoning through the average fusion of multi-stage low-rank parameters; S5. Input the test set and calculate the accuracy of the pre-trained image feature extraction model.

2. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 1, characterized in that: The visual branch Use ResNet or Transformer structure, text branch The Transformer structure is adopted; the fine-grained semantic prototype , , represents a set of real numbers, where N represents the number of fine-grained semantic prototypes, Represents the feature dimension.

3. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 2, characterized in that: In S22, the following logic is used to represent the input of the original training set image and the enhanced training set image into the visual branch of the CLIP model respectively: In the formula, Represents the uth original training set image The image features, Represents the u-th enhanced training set image The image features, Represents the original training set image The enhanced training set images after appearance enhancement are generated by the RandAug library. Represents the visual branch of the CLIP model; The probability score between the original training set images and the fine-grained semantic prototypes is calculated using the following logic: In the formula, express and The cosine similarity between express and The cosine similarity between represents exponential operation, represents the temperature coefficient, represents the probability score between the original training set image and the fine-grained semantic prototype, Representing fine-grained semantic prototypes Middle fine-grained semantic features, Representing fine-grained semantic prototypes Middle Fine-grained semantic features; The probability score between the enhanced training set images and the fine-grained semantic prototypes is calculated using the following logic: In the formula, express and The cosine similarity between express and The cosine similarity between Represents the probability score between the augmented training set images and the fine-grained semantic prototypes.

4. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 3, characterized in that: In S23, the following logic is used to represent the supervisory signal of the original training set image: In the formula, represents the supervisory signal of the original training set images, represents the Sinkhorn-Knopp optimal transmission algorithm, represents the probability distribution of the enhanced training set images in the fine-grained semantic prototype, ; The following logic is used to enhance the supervisory signal of the training set images: In the formula, represents the supervisory signal for enhancing the training set images, represents the probability distribution of the original training set images in the fine-grained semantic prototype, ; The global semantic score alignment loss is calculated using the following logic: In the formula, represents the global semantic score alignment loss, represents the KL loss, Indicates the use of measure The difference Indicates the use of measure difference.

5. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 4, characterized in that: The following logic is used in S24 to query the vector, key vector and value vector: In the formula, represents the query vector after mapping, , represents the key vector after mapping, , represents the value vector after mapping, , represents the dimension of the vector after mapping, represents the mth fine-grained semantic feature, , Represents the uth original training set image The local image features, , , represents a learnable mapping matrix; The following logic is used to extract the local semantic features corresponding to the fine-grained semantic features through the attention mechanism: In the formula, Representing fine-grained semantic features In the original training set images The local semantic features extracted are represents the transpose of the key vector, Indicates the use of attention mechanism, express function; The following logic is used to calculate the local semantic feature contrast loss: In the formula, represents the local semantic feature contrast loss, cos( )express and The cosine similarity between express and The cosine similarity between Representing fine-grained semantic features In the original training set images The extracted local semantic features, Representing fine-grained semantic features In the The local semantic features extracted from the original training set images, Indicates the size of the training set. Represents the temperature coefficient.

6. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 1, characterized in that: The S3 includes: S31, input the training set labels into Wikipedia or a large language model to obtain fine-grained text descriptions, and input them into the text branch of the CLIP model to obtain fine-grained text features of the category; S32, respectively calculating the cosine similarity between all fine-grained text features and each fine-grained semantic feature, if the cosine similarity is greater than 0.5, the corresponding fine-grained semantic feature is regarded as general knowledge, otherwise it is regarded as special knowledge, and a special knowledge image semantic prototype is obtained; S33, input the training set image into the visual branch of the CLIP model, obtain the image features of the training set image, calculate the similarity between the image features of the training set image and the semantic prototype of the dedicated knowledge image, obtain the dedicated knowledge image mask, and calculate the image features containing only the dedicated knowledge; S34. Input the training set labels into the text branch of the CLIP model to obtain the text features of the training set labels; input the training set images into the visual branch of the CLIP model with low-rank parameters to obtain the image features of the new training set images, and perform a product operation with the special knowledge image mask, and then construct a cross-entropy loss with the text features of the training set labels, and update the low-rank parameters through gradient backpropagation.

7. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 6, characterized in that: The following logic is used in S31 to obtain a fine-grained text description: In the formula, Indicates the category The detailed description is A fine-grained text description of represents the cth category, c , G represents the number of categories, represents a detailed description of category e obtained from Wikipedia or a large language model, , represents the number of detailed descriptions retrieved for each artificially set category; Use the following logic to obtain the fine-grained text features of this category: In the formula, Represents fine-grained text description Fine-grained text features.

8. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 7, characterized in that: In S33, the image features of the training set images are obtained using the following logic: In the formula, represents the training set image of the nth stage, Represents the training set images Image features, x, y represent the indexes of the horizontal and vertical positions of the training set images respectively; The similarity between the image features of the training set images and the semantic prototypes of the dedicated knowledge images is calculated using the following logic: try( )= In the formula, sim( ) represents the training set image Image features Similarity with the semantic prototype of the dedicated knowledge image, Represents the semantic features of dedicated knowledge images, , Representing Specialized Knowledge Image Semantic Prototypes , NS represents the number of dedicated image semantic prototypes, Representation calculation and The cosine similarity of The dedicated knowledge image mask is obtained using the following logic: In the formula, Represents a dedicated knowledge image mask; The image features containing only specialized knowledge are calculated using the following logic: In the formula, Represents image features that contain only specialized knowledge.

9. A method for continuous learning of a fault diagnosis model for substation equipment according to claim 8, characterized in that: In S34, the text features of the training set labels are calculated using the following logic: In the formula, Indicates the category The text description of the training set labels, Text features representing the training set labels; The following logic is used to calculate the probability that the image feature belongs to the cth category: In the formula, Represents the training set images The probability of belonging to the cth category, Representation calculation and The cosine similarity of Representation calculation and The cosine similarity of Indicates the category Text features of the training set labels; The cross entropy loss is constructed using the following logic: In the formula, represents the cross entropy loss, Represents the training set images If the true label belongs to the cth category, then ,otherwise 0.

10. A continuous learning system for a fault diagnosis model of a substation equipment, characterized in that: It includes training set processing module, knowledge extraction module, knowledge decoupling module, continuous learning module and test set testing module; The training set processing module is used to obtain a training set of substation equipment failures in a single stage, which is divided into a training set and a test set. The training set includes training set images and training set labels, and the test set includes test set images and test set labels. The training set images are randomly cropped, all training set images are scaled to a uniform size, and then numerical normalization is performed. The knowledge extraction module is used to establish a pre-trained image feature extraction model based on the Pytorch deep learning framework, wherein the model includes a CLIP model and a learnable fine-grained semantic prototype, implements fine-grained image semantic knowledge extraction based on global semantic score alignment and local semantic feature comparison, and performs gradient update on the fine-grained semantic prototype; A modeling unit for constructing a CLIP model and a learnable fine-grained semantic prototype; the CLIP model includes a visual branch and text branch ; The probability score unit is used to enhance the appearance of the original training set images. The original training set images and the enhanced training set images are respectively input into the visual branch of the CLIP model to calculate the probability scores between the original training set images and the fine-grained semantic prototypes, and the probability scores between the enhanced training set images and the fine-grained semantic prototypes. A global unit is used to obtain the supervision signal of the original training set image by transferring the probability score between the enhanced training set image and the fine-grained semantic prototype through the optimal transmission algorithm, and to calculate the global semantic score alignment loss; The local unit is used to map local image features using fine-grained semantic prototypes, calculate query vectors, key vectors, and value vectors, extract local semantic features corresponding to fine-grained semantic features through the attention mechanism, and calculate the local semantic feature contrast loss; The knowledge decoupling module is used to calculate the matching degree between fine-grained text features and fine-grained semantic features, divide the fine-grained semantic features into general knowledge and special knowledge, use low-rank parameters for learning and fusion, construct cross entropy loss, and update low-rank parameters through gradient backpropagation; The continuous learning module is used for the continuous learning process of substation equipment failures at different stages, and continuously updates the pre-trained image feature extraction model. After the training of each stage is completed, the dedicated knowledge low-rank parameters of the stage are obtained, and the low-rank parameters used in reasoning are obtained by averaging and fusing the low-rank parameters of multiple stages. The test set testing module is used to input a test set and calculate the accuracy of the pre-trained image feature extraction model.

Images