A method and system for multi-substation equipment defect identification based on federated learning

Abstract

A method and system for multi-substation equipment defect recognition based on federated learning, belonging to the technical field of image processing, solves the problem of feature offset caused by invisible client data in the process of substation equipment defect recognition; in the process of federated learning, the visual branch and text branch of the CLIP model do not need to be transmitted between the local end and the server end, and only the parameters of the text prompt words, encoder and decoder need to be transmitted to realize knowledge learning and knowledge fusion of different server ends. Compared with the CLIP model with a large number of parameters, the amount of transmission parameters is greatly reduced. Through multiple local and server-side cyclic training, a global model of the server end with the best effect is finally obtained. For the local end model, visual features of any category, especially unseen categories, can be generated, reducing the phenomenon of feature offset caused by invisible server-side data in the process of substation equipment defect recognition, and improving the generalization ability of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a method and system for identifying defects in multi-substation equipment based on federated learning. Background Art

[0002] In recent years, visual language pre-training models represented by the CLIP model have demonstrated powerful processing capabilities in visual tasks by pre-training on a large amount of visual-text pairing data. They have good zero-sample recognition capabilities for conventional natural image processing. For most semantic categories, the pre-trained models already have preliminary recognition capabilities and only require a small amount of parameter fine-tuning to adapt to the target dataset and handle downstream tasks.

[0003] The invention patent with publication number CN115829028A discloses a method for processing multimodal federated learning tasks. Through linear embedding of multimodal data, modal transfer federated training based on linear embedding, and federated learning task training using the same-mode embedding results, it supports federated learning training for participants with different data modalities to obtain a better model than local training.

[0004] The invention patent with publication number CN118261236A discloses a federated learning method for multimodal data. It proposes an improvement plan to address the problem that traditional federated learning methods lack robustness to modal missing situations. The server will assign client weights based on three factors, including the amount of client data, the proportion of client modal missingness, and the contribution of the missing modality to the task, to reduce the impact of modal missing clients on the global model.

[0005] The existing technologies mainly design pre-training models under the traditional CNN model or Transformer model structure. For specific downstream tasks, such as when dealing with multi-substation equipment defect identification tasks, it is necessary to collect data from multiple substations. The traditional centralized learning method needs to collect the original data of each substation, resulting in the risk of leakage of sensitive data during transmission and storage; even if some existing technologies adopt distributed learning, they do not propose an effective feature fusion mechanism for the problem of data invisibility in a multi-client environment, and the data invisibility between clients exacerbates the problem of visual feature offset. In addition, although the CLIP model has a strong zero-sample capability, due to its large number of parameters, it is difficult to directly use it in a federated learning environment for efficient local knowledge learning and global parameter aggregation. Therefore, it is particularly important to realize the knowledge learning and knowledge fusion of different clients with a small number of parameters in the visual language pre-training model, and to solve the problem of feature offset caused by the invisibility of client data. Summary of the invention

[0006] The technical problem to be solved by the present invention is to solve the problem of feature deviation caused by invisible client data during the defect identification process of substation equipment.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] A method for identifying defects in multi-substation equipment based on federated learning, comprising the following steps:

[0009] S1. Obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0010] S2. Constructing a visual feature generation model based on an encoder-decoder structure on the local side to construct a learnable text prompt word, wherein the visual feature generation model includes a CLIP model, an encoder and a decoder, and initializing the parameters of the encoder and the decoder;

[0011] S3, fix the text prompt word at the local end, input the training set image into the visual branch of the CLIP model and output the original visual features as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual features, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, and calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model;

[0012] S4. Fix the parameters of the visual feature generation model on the local side, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample the standard normal score to form the second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient backpropagation;

[0013] S5. Upload the text prompt words, encoder and decoder parameters of the local end to the server end, build a global model based on federated learning on the server end, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss;

[0014] S6. Distribute the parameters of the global model to the local end, repeat S3 to S5 until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model.

[0015] Furthermore, the CLIP model in S2 includes a visual branch and text branch , visual branch Adopt ViT model structure, accept image input, text branch Adopts Transformer structure and accepts text input containing categories;

[0016] The following logic is used to represent the visual branch Accepts image input:

[0017]

[0018] In the formula, represents the image input, Represents the original visual features;

[0019] The text branches are represented using the following logic Accepts text input containing categories:

[0020]

[0021]

[0022] In the formula, Represents the text description of category c. represents the name of the cth category, , represents the number of categories, , Indicates text prompt words, Represents the text features of category c.

[0023] Furthermore, the following logic is used in S3 to represent the internal reconstruction visual features:

[0024]

[0025] In the formula, represents the internal reconstructed visual features of category c, It indicates the residual feature output by the decoder after accepting the first latent variable as input. , Represents a decoder, represents the first latent variable;

[0026] The reconstruction loss is calculated using the following logic:

[0027]

[0028] In the formula, represents the reconstruction loss, represents the mean square error loss function, Represents the calculation of raw visual features and internal reconstruction of visual features The mean square error loss is represents the KL loss, Indicates that the first latent variable is c. The probability distribution of each element in is Follows standard normal distribution The prior distribution of Indicates the use of encoder output measure The difference , Represents an encoder.

[0029] Furthermore, the following logic is used in S3 to generate visual features:

[0030]

[0031] In the formula, represents the generated visual features of category c, It indicates the residual feature output by the decoder after accepting the second latent variable as input. , represents the second latent variable;

[0032] The following logic is used to calculate the category scores of the training visual features and text features of different categories through the softmax function:

[0033]

[0034]

[0035] In the formula, Represents training visual features The probability of belonging to the cth category, represents the i-th training visual feature, represents exponential operation, express and The cosine similarity between represents L2 normalization, express and The cosine similarity between represents the text features of category j, represents the temperature coefficient;

[0036] The cross entropy loss is constructed using the following logical representation:

[0037]

[0038] In the formula, represents the cross entropy loss, N represents the number of training visual features, represents the category of the i-th training set label, Represents model predictions Belongs to category probability.

[0039] Further, the S5 comprises the following steps:

[0040] S51, averaging the parameters of the text prompt words, encoder and decoder of the local end on the server end to obtain the text prompt words, encoder and decoder of the global model;

[0041] S52, performing knowledge distillation on the encoder and decoder of the global model on the server side, randomly sampling in the standard normal score to form a third latent variable, using the text prompt words of the global model to obtain the text features of the global model, calculating the reconstructed visual features of the local side and the fourth latent variable of the local side, and calculating the reconstructed visual features of the server side and the fifth latent variable of the server side;

[0042] S53. Calculate the feature distillation loss and latent variable distillation loss on the server side and construct the total loss on the server side.

[0043] Furthermore, the text prompt words, encoder and decoder of the global model are represented by the following logic in S51:

[0044]

[0045]

[0046]

[0047] In the formula, The nth parameter of the text prompt word representing the global model, ,make , represents the text prompt word of the global model, M represents the number of local visual feature generation models, Indicates the nth parameter of the mth local text prompt word. , represents the encoder of the global model, represents the mth local encoder, represents the decoder of the global model, Indicates the mth local decoder.

[0048] Furthermore, the following logic is used in S52 to calculate the reconstructed visual features of the local end:

[0049]

[0050]

[0051]

[0052] In the formula, Indicates the local decoder The residual features output after accepting the third latent variable input, represents the third latent variable, represents a text description of category c of the global model, Represents the reconstructed visual features of category c at the mth local end;

[0053] The fourth hidden variable on the local side is calculated using the following logic:

[0054]

[0055] In the formula, represents the fourth latent variable, Indicates the mth local encoder.

[0056] Furthermore, the reconstructed visual features of the server are calculated using the following logic in S52:

[0057]

[0058]

[0059] In the formula, Decoder representing the global model The residual features output after accepting the third latent variable input, represents the reconstructed visual features of category c on the server side;

[0060] The fifth hidden variable on the server side is calculated using the following logic:

[0061]

[0062] In the formula, represents the fifth latent variable.

[0063] Furthermore, the characteristic distillation loss is calculated using the following logic in S53:

[0064]

[0065] In the formula, represents the feature distillation loss;

[0066] The following logic is used to calculate the latent variable distillation loss:

[0067]

[0068] In the formula, represents the latent variable distillation loss;

[0069] The total loss on the server side is calculated using the following logic:

[0070]

[0071] In the formula, Represents the total loss on the server side.

[0072] A multi-substation equipment defect identification system based on federated learning, comprising a data processing module, a local model building module, a local parameter updating module, a local text prompt word training module, a server model building module and a server model training module;

[0073] The data processing module is used to obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, the training set includes training set images and training set labels, the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0074] The local end model building module is used to construct a visual feature generation model based on an encoder-decoder structure at the local end, construct a learnable text prompt word, and the visual feature generation model includes a CLIP model, an encoder and a decoder, and initialize the parameters of the encoder and the decoder;

[0075] The local parameter updating module fixes the text prompt word at the local end, inputs the training set image into the visual branch of the CLIP model and outputs the original visual feature as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual feature, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model;

[0076] The local text prompt word training module is used to fix the parameters of the visual feature generation model at the local end, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample in the standard normal score to form a second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient back propagation;

[0077] The server-side model building module is used to upload the text prompt words, encoder and decoder parameters of the local side to the server side, build a global model based on federated learning on the server side, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss;

[0078] The server-side model training module is used to distribute the parameters of the global model to the local side, repeatedly execute the local-side parameter update module to the server-side model building module until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model.

[0079] The advantages of the present invention are:

[0080] (1) In the federated learning process, the visual branch and text branch of the CLIP model do not need to be transmitted between the local end and the server end. Only the parameters of the text prompt words, encoder and decoder need to be transmitted to achieve knowledge learning and knowledge fusion on different server ends. Compared with the CLIP model with a large number of parameters, the amount of transmitted parameters is greatly reduced.

[0081] (2) The present invention assigns parameters of the global model by averaging the parameters of the local model, integrates the knowledge of different local models into the global model on the server side, calculates the distillation loss based on the reconstructed visual features and latent variables, trains the global model, and redistributes the text prompt words, encoder and decoder parameters of the global model to the local side after the training. Through multiple local and server-side cyclic training, the global model on the server side with the best effect is finally obtained. For the local model, visual features of any category, especially unseen categories, can be generated, reducing the phenomenon of feature offset caused by invisible server-side data in the process of substation equipment defect identification, thereby improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a flow chart of a method for identifying defects in multi-substation equipment based on federated learning according to the first embodiment of the present invention;

[0083] Figure 2is a schematic diagram of updating the parameters of the local visual feature generation model according to the first embodiment of the present invention;

[0084] Figure 3 is a schematic diagram of updating parameters of a local text prompt word according to the first embodiment of the present invention;

[0085] Figure 4 It is a schematic diagram of a federated learning process between a local end and a server end in Embodiment 1 of the present invention;

[0086] Figure 5 Schematic diagram of text features of a global model generated by a server in Embodiment 1 of the present invention;

[0087] Figure 6 It is a schematic diagram of the knowledge distillation of the encoder and the decoder performed by the server side in the first embodiment of the present invention. DETAILED DESCRIPTION

[0088] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0089] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0090] Embodiment 1

[0091] like Figure 1 Specifically, a method for identifying defects in multi-substation equipment based on federated learning is disclosed, comprising the following steps:

[0092] S1. Obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0093] In this embodiment, the multi-substation equipment defect identification process is divided into multiple stages, and sample data of multiple substation equipment defects are obtained. 80% of the sample data are divided into a training set, and 20% of the sample data are divided into a test set. The training set images are processed separately. The training set of multi-substation equipment defect identification in a single stage is taken as an example to illustrate. The training set images are randomly cropped, and the size of all training set images is scaled to 512×1024, and then numerical normalization is performed.

[0094] S2. Constructing a visual feature generation model based on an encoder-decoder structure on the local side to construct a learnable text prompt word, wherein the visual feature generation model includes a CLIP model, an encoder and a decoder, and initializing the parameters of the encoder and the decoder;

[0095] In this embodiment, the CLIP model is trained based on large-scale image-text pairs by contrastive learning. The CLIP model includes a visual branch and text branch , where the visual branch Using ResNet or Transformer structure, the visual branch in this embodiment Adopt ViT model structure, text branch Adopt Transformer structure.

[0096] The visual branch of the CLIP model Accepts image input, such as ,in represents the image input, Represents the original visual features.

[0097] The text branch of the CLIP model Accepts text input containing categories, such as text description , the text description is represented as a table containing specific semantic categories of images, where From the number of Category collection Select from.

[0098] In this embodiment, the visual branch of the CLIP model and text branch Fixed parameters are used, and in order to make the model learn local data knowledge, the text branch The text description of the input is given by Change to ,make , Indicates text prompt words, represents the learnable parameters updated by gradient back propagation during the text prompt word training process. In this embodiment, the text description is input into the text branch of the CLIP model to obtain the corresponding text features using the following logic:

[0099]

[0100]

[0101] In the formula, represents the name of the cth category, , Represents the text description of category c. represents the text features of category c, , Indicates a text prompt word.

[0102] S3, fix the text prompt word at the local end, input the training set image into the visual branch of the CLIP model and output the original visual features as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual features, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, and calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model;

[0103] In this embodiment, if Figure 2 As shown in Figure 2, the text prompt word is fixed on the local side, and the training set image is input into the visual branch of the CLIP model. And output the original visual features , the original visual features As the input of encoder E, encoder E outputs the first latent variable , the decoder Q accepts the first latent variable Z as input and outputs the residual feature , add the output of decoder Q to the text features to obtain the internal reconstructed visual features, calculate the reconstruction loss, and use the reconstruction loss to update the parameters of the visual feature generation model.

[0104] In this embodiment, the following logic is used to represent the internal reconstruction visual features:

[0105]

[0106] In the formula, represents the internal reconstructed visual features of category c.

[0107] In this embodiment, the decoder includes two fully connected layers at the beginning and the end and a ReLU activation layer in the middle. Based on the learning method of the autoencoder, the reconstruction loss is calculated using the following logic, and the parameters of the visual feature generation model are updated using the reconstruction loss:

[0108]

[0109] In the formula, represents the reconstruction loss, represents the mean square error loss function, Represents the calculation of raw visual features and internal reconstruction of visual features The mean square error loss is represents the KL loss, Indicates that the first latent variable is c. The probability distribution of each element in is Follows standard normal distribution The prior distribution of Indicates the use of encoder output measure The difference , Represents an encoder.

[0110] In this embodiment, the reconstructed visual features are obtained by adding the output of the decoder to the text features. The decoder is equivalent to learning the residual from a text feature to a visual feature. In this embodiment, the decoder is used to generate visual features of all semantic label categories, and a reconstruction loss between the original visual features and the reconstructed visual features is constructed. The loss is used to update the parameters of the encoder E and the decoder Q by gradient updating, so as to achieve local encoder. and decoder Training and updating of visual feature generation model.

[0111] S4. Fix the parameters of the visual feature generation model on the local side, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample the standard normal score to form the second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient backpropagation;

[0112] In this embodiment, if Figure 3 As shown in Figure 2, the parameters of the visual feature generation model are fixed locally to construct a learnable text prompt word. , the text branch of the CLIP model Output the text features of each category and randomly sample the standard normal score to form the second latent variable , the output of decoder Q is added to the text feature to generate the visual feature, which is represented by the following logic:

[0113]

[0114] In the formula, represents the generated visual features of category c, Indicates that the decoder accepts the second latent variable As the residual feature of the output after input, .

[0115] In this embodiment, visual features are generated With the original visual features Together they form a complete training visual feature , , used for text prompt words Parameter learning.

[0116] In this embodiment, the following logic is used to calculate the training visual features through the softmax function: Category scores of text features with different categories:

[0117]

[0118]

[0119] In the formula, Represents training visual features The probability of belonging to the cth category, represents the i-th training visual feature, represents exponential operation, express and The cosine similarity between represents L2 normalization, express and The cosine similarity between represents the text features of category j, Indicates the temperature coefficient. In this embodiment Take 0.01.

[0120] In this embodiment, based on the above-mentioned local self-encoding framework, after fusion of the server-side model, the encoder E and decoder Q trained on more categories and data can be obtained, so in the process of learning local knowledge, data of all categories can be generated.

[0121] In this embodiment, the text prompt word The parameter learning of can be updated by constructing the cross entropy loss. This embodiment uses the following logic to construct the cross entropy loss:

[0122]

[0123] In the formula, represents the cross entropy loss, N represents the number of training visual features, represents the category of the i-th training set label, Represents model predictions Belongs to category probability.

[0124] In this embodiment, during the federated learning process, the visual branch and text branch of the CLIP model do not need to be transmitted between the local end and the server end. Only the parameters of the text prompt words, encoder and decoder need to be transmitted to achieve knowledge learning and knowledge fusion of different server ends. Compared with the CLIP model with a large number of parameters, the amount of transmission parameters is greatly reduced.

[0125] S5. Upload the text prompt words, encoder and decoder parameters of the local end to the server end, build a global model based on federated learning on the server end, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss;

[0126] like Figure 4 As shown, the S5 comprises the following steps:

[0127] S51, averaging the parameters of the text prompt words, encoder and decoder of the local end on the server end to obtain the text prompt words, encoder and decoder of the global model;

[0128] In this embodiment, it is assumed that there are M local terminals for model training. After the training is completed, the parameters of the M local terminal models are transmitted to the server, including the text prompt word , encoder E and decoder Q; subscript the text prompt words, The text hint word representing the first local model.

[0129] In this embodiment, the text prompt words, encoder and decoder of the global model are represented by the following logic:

[0130]

[0131]

[0132]

[0133] In the formula, The nth parameter of the text prompt word representing the global model, ,make , represents the text prompt word of the global model, M represents the number of local visual feature generation models, Indicates the nth parameter of the mth local text prompt word. , represents the encoder of the global model, represents the mth local encoder, represents the decoder of the global model, Indicates the mth local decoder.

[0134] S52, performing knowledge distillation on the encoder and decoder of the global model on the server side, randomly sampling in the standard normal score to form a third latent variable, using the text prompt words of the global model to obtain the text features of the global model, calculating the reconstructed visual features of the local side and the fourth latent variable of the local side, and calculating the reconstructed visual features of the server side and the fifth latent variable of the server side;

[0135] like Figure 6 As shown, in this embodiment, the autoencoder framework is distilled from the local model to the global model, and the distillation process from the mth local visual feature generation model to the server-side global model is used as an example to illustrate, and the third latent variable is randomly sampled in the standard normal score , after the mth local decoder Output residual features, such as Figure 5 As shown, the text prompt words using the global model The text branch of the CLIP model is input to obtain the text features of the global model, and the local reconstructed visual features are calculated in combination with the residual features. This embodiment uses the following logic to calculate the local reconstructed visual features:

[0136]

[0137]

[0138]

[0139] In the formula, Indicates the local decoder The residual features output after accepting the third latent variable input, represents the third latent variable, represents a text description of category c of the global model, Represents the reconstructed visual features of category c at the mth local end.

[0140] In this embodiment, Through the mth local encoder Get the fourth hidden variable This embodiment uses the following logic to calculate the fourth hidden variable on the local side: :

[0141]

[0142] In the formula, represents the fourth latent variable, Indicates the mth local encoder.

[0143] In this embodiment, for the global model, the third latent variable Input to the decoder of the global model Combine the text features of the global model to calculate the reconstructed visual features on the server side:

[0144]

[0145]

[0146] In the formula, Decoder representing the global model The residual features output after accepting the third latent variable input, Represents the reconstructed visual features of category c on the server side.

[0147] In this embodiment, Input to the encoder of the global model , calculate the fifth hidden variable on the server side :

[0148]

[0149] In the formula, represents the fifth latent variable.

[0150] S53, calculating the feature distillation loss and the latent variable distillation loss on the server side, and constructing the total loss on the server side;

[0151] In this embodiment, if Figure 6 As shown, this embodiment calculates the distillation loss based on the reconstructed visual features and latent variables, and uses the following logic to calculate the feature distillation loss:

[0152]

[0153] In the formula, represents the feature distillation loss.

[0154] This example uses the following logic to calculate the latent variable distillation loss:

[0155]

[0156] In the formula, represents the latent variable distillation loss.

[0157] This embodiment uses the following logic to calculate the total loss on the server side:

[0158]

[0159] In the formula, Represents the total loss on the server side.

[0160] S6. Distribute the parameters of the global model to the local end, repeat S3 to S5 until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model;

[0161] In this embodiment, the parameters of the global model are assigned by averaging the parameters of the local model, the knowledge of different local models is integrated into the global model of the server, the distillation loss is calculated based on the reconstructed visual features and latent variables, the global model is trained by the above loss, the text prompt words, encoder and decoder parameters of the global model after the training are redistributed to the local end, the local visual feature generation model is trained, the local text prompt words, encoder and decoder parameters are uploaded to the server, the server-side global model is trained, and the parameters of the global model are distributed to the local end again, and through multiple local and server-side cyclic training, the global model of the server with the best effect is finally obtained.

[0162] Embodiment 2

[0163] The present invention also discloses a multi-substation equipment defect identification system based on federated learning, including a data processing module, a local end model building module, a local end parameter updating module, a local end text prompt word training module, a server end model building module and a server end model training module;

[0164] The data processing module is used to obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, the training set includes training set images and training set labels, the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing;

[0165] In this embodiment, the multi-substation equipment defect identification process is divided into multiple stages, and sample data of multiple substation equipment defects are obtained. 80% of the sample data are divided into a training set, and 20% of the sample data are divided into a test set. The training set images are processed separately. The training set of multi-substation equipment defect identification in a single stage is taken as an example to illustrate. The training set images are randomly cropped, and the size of all training set images is scaled to 512×1024, and then numerical normalization is performed.

[0166] The local end model building module is used to construct a visual feature generation model based on an encoder-decoder structure at the local end, construct a learnable text prompt word, and the visual feature generation model includes a CLIP model, an encoder and a decoder, and initialize the parameters of the encoder and the decoder;

[0167] In this embodiment, the CLIP model is trained based on large-scale image-text pairs by contrastive learning. The CLIP model includes a visual branch and text branch , where the visual branch Using ResNet or Transformer structure, the visual branch in this embodiment Adopt ViT model structure, text branch Adopt Transformer structure.

[0168] The visual branch of the CLIP model Accepts image input, such as ,in represents the image input, Represents the original visual features.

[0169] The text branch of the CLIP model Accepts text input containing categories, such as text description , the text description is represented as a table containing specific semantic categories of images, where From the number of Category collection Select from.

[0170] In this embodiment, the visual branch of the CLIP model and text branch Fixed parameters are used, and in order to make the model learn local data knowledge, the text branch The text description of the input is given by Change to ,make , Indicates text prompt words, represents the learnable parameters updated by gradient back propagation during the text prompt word training process. In this embodiment, the text description is input into the text branch of the CLIP model to obtain the corresponding text features using the following logic:

[0171]

[0172]

[0173] In the formula, represents the name of the cth category, , Represents the text description of category c. represents the text features of category c, , Indicates a text prompt word.

[0174] The local parameter updating module fixes the text prompt word at the local end, inputs the training set image into the visual branch of the CLIP model and outputs the original visual feature as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual feature, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model;

[0175] In this embodiment, the text prompt words are fixed at the local end, and the training set images are input into the visual branch of the CLIP model. And output the original visual features , the original visual features As the input of encoder E, encoder E outputs the first latent variable , the decoder Q accepts the first latent variable Z as input and outputs the residual feature , add the output of decoder Q to the text features to obtain the internal reconstructed visual features, calculate the reconstruction loss, and use the reconstruction loss to update the parameters of the visual feature generation model.

[0176] In this embodiment, the following logic is used to represent the internal reconstruction visual features:

[0177]

[0178] In the formula, represents the internal reconstructed visual features of category c.

[0179] In this embodiment, the decoder includes two fully connected layers at the beginning and the end and a ReLU activation layer in the middle. Based on the learning method of the autoencoder, the reconstruction loss is calculated using the following logic, and the parameters of the visual feature generation model are updated using the reconstruction loss:

[0180]

[0181] In the formula, represents the reconstruction loss, represents the mean square error loss function, Represents the calculation of raw visual features and internal reconstruction of visual features The mean square error loss is represents the KL loss, Indicates that the first latent variable is c. The probability distribution of each element in is Follows standard normal distribution The prior distribution of Indicates the use of encoder output measure The difference , Represents an encoder.

[0182] In this embodiment, the reconstructed visual features are obtained by adding the output of the decoder to the text features. The decoder is equivalent to learning the residual from a text feature to a visual feature. In this embodiment, the decoder is used to generate visual features of all semantic label categories, and a reconstruction loss between the original visual features and the reconstructed visual features is constructed. The loss is used to update the parameters of the encoder E and the decoder Q by gradient updating, so as to achieve local encoder. and decoder Training and updating the visual feature generation model.

[0183] The local text prompt word training module is used to fix the parameters of the visual feature generation model at the local end, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample in the standard normal score to form a second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient back propagation;

[0184] In this embodiment, the parameters of the visual feature generation model are fixed on the local side to construct a learnable text prompt word , the text branch of the CLIP model Output the text features of each category and randomly sample the standard normal score to form the second latent variable , the output of decoder Q is added to the text feature to generate the visual feature, which is represented by the following logic:

[0185]

[0186] In the formula, represents the generated visual features of category c, Indicates that the decoder accepts the second latent variable As the residual feature of the output after input, .

[0187] In this embodiment, visual features are generated With the original visual features Together they form a complete training visual feature , , used for text prompt words Parameter learning.

[0188] In this embodiment, the following logic is used to calculate the training visual features through the softmax function: Category scores of text features with different categories:

[0189]

[0190]

[0191] In the formula, Represents training visual features The probability of belonging to the cth category, represents the i-th training visual feature, represents exponential operation, express and The cosine similarity between represents L2 normalization, express and The cosine similarity between represents the text features of category j, Indicates the temperature coefficient. In this embodiment Take 0.01.

[0192] In this embodiment, based on the above local self-encoding framework, after the fusion of the server-side model, an encoder trained on more categories and data can be obtained. and decoder , so in the process of learning local knowledge, data of all categories can be generated.

[0193] In this embodiment, the text prompt word The parameter learning of can be updated by constructing the cross entropy loss. This embodiment uses the following logic to construct the cross entropy loss:

[0194]

[0195] In the formula, represents the cross entropy loss, N represents the number of training visual features, represents the category of the i-th training set label, Represents model predictions Belongs to category probability.

[0196] In this embodiment, during the federated learning process, the visual branch and text branch of the CLIP model do not need to be transmitted between the local end and the server end. Only the parameters of the text prompt words, encoder and decoder need to be transmitted to achieve knowledge learning and knowledge fusion of different server ends. Compared with the CLIP model with a large number of parameters, the amount of transmission parameters is greatly reduced.

[0197] The server-side model building module is used to upload the text prompt words, encoder and decoder parameters of the local side to the server side, build a global model based on federated learning on the server side, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss;

[0198] The server-side model building module includes a parameter unit, a knowledge distillation unit and a loss unit;

[0199] A parameter unit, used for averaging the parameters of the text prompt words, encoders and decoders of the local end on the server end to obtain the text prompt words, encoders and decoders of the global model;

[0200] In this embodiment, it is assumed that there are M local terminals for model training. After the training is completed, the parameters of the M local terminal models are transmitted to the server, including the text prompt word , encoder E and decoder Q; subscript the text prompt words, The text hint word representing the first local model.

[0201] In this embodiment, the text prompt words, encoder and decoder of the global model are represented by the following logic:

[0202]

[0203]

[0204]

[0205] In the formula, The nth parameter of the text prompt word representing the global model, ,make , represents the text prompt word of the global model, M represents the number of local visual feature generation models, Indicates the nth parameter of the mth local text prompt word. , represents the encoder of the global model, represents the mth local encoder, represents the decoder of the global model, Indicates the mth local decoder.

[0206] A knowledge distillation unit is used to perform knowledge distillation on the encoder and decoder of the global model on the server side, randomly sample the standard normal score to form a third latent variable, use the text prompt words of the global model to obtain the text features of the global model, calculate the reconstructed visual features on the local side and the fourth latent variables on the local side, and calculate the reconstructed visual features on the server side and the fifth latent variables on the server side;

[0207] In this embodiment, the autoencoder framework is distilled from the local model to the global model. The distillation process from the mth local visual feature generation model to the server-side global model is used as an example to illustrate the third latent variable formed by random sampling in the standard normal score. , after the mth local decoder Output residual features, such as Figure 5 As shown, the text prompt words using the global model The text branch of the CLIP model is input to obtain the text features of the global model, and the local reconstructed visual features are calculated in combination with the residual features. This embodiment uses the following logic to calculate the local reconstructed visual features:

[0208]

[0209]

[0210]

[0211] In the formula, Indicates the local decoder The residual features output after accepting the third latent variable input, represents the third latent variable, represents a text description of category c of the global model, Represents the reconstructed visual features of category c at the mth local end.

[0212] In this embodiment, Through the mth local encoder Get the fourth hidden variable This embodiment uses the following logic to calculate the fourth hidden variable on the local side: :

[0213]

[0214] In the formula, represents the fourth latent variable, Indicates the mth local encoder.

[0215] In this embodiment, for the global model, the third latent variable Input to the decoder of the global model Combine the text features of the global model to calculate the reconstructed visual features on the server side:

[0216]

[0217]

[0218] In the formula, Decoder representing the global model The residual features output after accepting the third latent variable input, Represents the reconstructed visual features of category c on the server side.

[0219] In this embodiment, Input to the encoder of the global model , calculate the fifth hidden variable on the server side :

[0220]

[0221] In the formula, represents the fifth latent variable.

[0222] The loss unit is used to calculate the feature distillation loss and latent variable distillation loss on the server side and construct the total loss on the server side;

[0223] In this embodiment, this embodiment calculates the distillation loss based on the reconstructed visual features and latent variables, and uses the following logic to calculate the feature distillation loss:

[0224]

[0225] In the formula, represents the feature distillation loss.

[0226] This example uses the following logic to calculate the latent variable distillation loss:

[0227]

[0228] In the formula, represents the latent variable distillation loss.

[0229] This embodiment uses the following logic to calculate the total loss on the server side:

[0230]

[0231] In the formula, Represents the total loss on the server side.

[0232] The server-side model training module is used to distribute the parameters of the global model to the local side, repeatedly execute the local-side parameter update module to the server-side model building module until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model.

[0233] In this embodiment, the parameters of the global model are assigned by averaging the parameters of the local model, the knowledge of different local models is integrated into the global model of the server, the distillation loss is calculated based on the reconstructed visual features and latent variables, the global model is trained by the above loss, the text prompt words, encoder and decoder parameters of the global model after the training are redistributed to the local end, the local visual feature generation model is trained, the local text prompt words, encoder and decoder parameters are uploaded to the server, the server-side global model is trained, and the parameters of the global model are distributed to the local end again, and through multiple local and server-side cyclic training, the global model of the server with the best effect is finally obtained.

[0234] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying defects in multi-substation equipment based on federated learning, characterized in that: The following steps are involved: S1. Obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, wherein the training set includes training set images and training set labels, and the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing; S2. Constructing a visual feature generation model based on an encoder-decoder structure on the local side to construct a learnable text prompt word, wherein the visual feature generation model includes a CLIP model, an encoder and a decoder, and initializing the parameters of the encoder and the decoder; S3, fix the text prompt word at the local end, input the training set image into the visual branch of the CLIP model and output the original visual features as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual features, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, and calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model; S4. Fix the parameters of the visual feature generation model on the local side, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample the standard normal score to form the second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient backpropagation; S5. Upload the text prompt words, encoder and decoder parameters of the local end to the server end, build a global model based on federated learning on the server end, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss; S6. Distribute the parameters of the global model to the local end, repeat S3 to S5 until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model.

2. A method for identifying defects in multi-substation equipment based on federated learning according to claim 1, characterized in that: The CLIP model in S2 includes a visual branch and text branch , visual branch Adopt ViT model structure, accept image input, text branch Adopts Transformer structure and accepts text input containing categories; The following logic is used to represent the visual branch Accepts image input: In the formula, represents the image input, Represents the original visual features; The text branches are represented using the following logic Accepts text input containing categories: In the formula, Represents the text description of category c. represents the name of the cth category, , represents the number of categories, , Indicates text prompt words, Represents the text features of category c.

3. A method for identifying defects in multi-substation equipment based on federated learning according to claim 2, characterized in that: The following logic is used in S3 to represent the internal reconstruction visual features: In the formula, represents the internal reconstructed visual features of category c, It indicates the residual feature output by the decoder after accepting the first latent variable as input. , Represents a decoder, represents the first latent variable; The reconstruction loss is calculated using the following logic: In the formula, represents the reconstruction loss, represents the mean square error loss function, Represents the calculation of raw visual features and internal reconstruction of visual features The mean square error loss is represents the KL loss, Indicates that the first latent variable is c. The probability distribution of each element in is Follows standard normal distribution The prior distribution of Indicates the use of encoder output measure The difference , Indicates the encoder.

4. A method for identifying defects in multi-substation equipment based on federated learning according to claim 3, characterized in that: The following logic is used in S3 to generate visual features: In the formula, represents the generated visual features of category c, It indicates the residual feature output by the decoder after accepting the second latent variable as input. , represents the second latent variable; The following logic is used to calculate the category scores of the training visual features and text features of different categories through the softmax function: In the formula, Represents training visual features The probability of belonging to the cth category, represents the i-th training visual feature, represents exponential operation, express and The cosine similarity between represents L2 normalization, express and The cosine similarity between represents the text features of category j, represents the temperature coefficient; The cross entropy loss is constructed using the following logical representation: In the formula, represents the cross entropy loss, N represents the number of training visual features, represents the category of the i-th training set label, Represents model predictions Belongs to category probability.

5. A method for identifying defects in multi-substation equipment based on federated learning according to claim 4, characterized in that: The S5 comprises the following steps: S51, averaging the parameters of the text prompt words, encoder and decoder of the local end on the server end to obtain the text prompt words, encoder and decoder of the global model; S52, performing knowledge distillation on the encoder and decoder of the global model on the server side, randomly sampling in the standard normal score to form a third latent variable, using the text prompt words of the global model to obtain the text features of the global model, calculating the reconstructed visual features of the local side and the fourth latent variable of the local side, and calculating the reconstructed visual features of the server side and the fifth latent variable of the server side; S53. Calculate the feature distillation loss and latent variable distillation loss on the server side and construct the total loss on the server side.

6. A method for identifying defects in multi-substation equipment based on federated learning according to claim 5, characterized in that: The text prompt words, encoder and decoder of the global model are represented by the following logic in S51: In the formula, The nth parameter of the text prompt word representing the global model, ,make , represents the text prompt word of the global model, M represents the number of local visual feature generation models, Indicates the nth parameter of the mth local text prompt word. , represents the encoder of the global model, represents the mth local encoder, represents the decoder of the global model, Indicates the mth local decoder.

7. A method for identifying defects in multi-substation equipment based on federated learning according to claim 6, characterized in that: In S52, the following logic is used to calculate the reconstructed visual features of the local end: In the formula, Indicates the local decoder The residual features output after accepting the third latent variable input, represents the third latent variable, represents a text description of category c of the global model, Represents the reconstructed visual features of category c at the mth local end; The fourth hidden variable on the local side is calculated using the following logic: In the formula, represents the fourth latent variable, Indicates the mth local encoder.

8. A method for identifying defects in multi-substation equipment based on federated learning according to claim 7, characterized in that: In S52, the following logic is used to calculate the reconstructed visual features on the server side: In the formula, Decoder representing the global model The residual features output after accepting the third latent variable input, represents the reconstructed visual features of category c on the server side; The fifth hidden variable on the server side is calculated using the following logic: In the formula, represents the fifth latent variable.

9. A method for identifying defects in multi-substation equipment based on federated learning according to claim 8, characterized in that: In S53, the characteristic distillation loss is calculated using the following logic: In the formula, represents the feature distillation loss; The following logic is used to calculate the latent variable distillation loss: In the formula, represents the latent variable distillation loss; The total loss on the server side is calculated using the following logic: In the formula, Represents the total loss on the server side.

10. A multi-substation equipment defect identification system based on federated learning, characterized in that: It includes a data processing module, a local model building module, a local parameter updating module, a local text prompt word training module, a server model building module and a server model training module; The data processing module is used to obtain sample data of multiple substation equipment defects, divide them into a training set and a test set, the training set includes training set images and training set labels, the test set includes test set images and test set labels, randomly crop the training set images, scale all training set images to a uniform size, and then perform numerical normalization processing; The local end model building module is used to construct a visual feature generation model based on an encoder-decoder structure at the local end, construct a learnable text prompt word, and the visual feature generation model includes a CLIP model, an encoder and a decoder, and initialize the parameters of the encoder and the decoder; The local parameter updating module fixes the text prompt word at the local end, inputs the training set image into the visual branch of the CLIP model and outputs the original visual feature as the input of the encoder, so that the encoder outputs the first latent variable, the decoder accepts the first latent variable as input, outputs the residual feature, adds the output of the decoder to the text feature, obtains the internal reconstructed visual feature, calculates the reconstruction loss, and uses the reconstruction loss to update the parameters of the visual feature generation model; The local text prompt word training module is used to fix the parameters of the visual feature generation model at the local end, input the text prompt word into the text branch of the CLIP model and output the text features of each category, randomly sample in the standard normal score to form a second latent variable, add the output of the decoder to the text feature to obtain the generated visual feature, and combine the original visual feature to form the training visual feature, calculate the category scores of the training visual feature and the text features of different categories through the softmax function, construct the cross entropy loss, and update the parameters of the text prompt word by gradient back propagation; The server-side model building module is used to upload the text prompt words, encoder and decoder parameters of the local side to the server side, build a global model based on federated learning on the server side, calculate the text prompt words, encoder and decoder parameters of the global model, and build feature distillation loss and latent variable distillation loss; The server-side model training module is used to distribute the parameters of the global model to the local side, repeatedly execute the local-side parameter update module to the server-side model building module until the maximum number of iterations is reached, input the test set, and calculate the accuracy of the global model.

Images