Substation defect detection model fine tuning method based on cascade comparative learning

Abstract

The invention discloses a transformer substation defect detection model fine tuning method based on cascade comparative learning, and the method comprises the following steps: 1, model pre-training: carrying out the pre-training of a model on an ImageNet sample image; 2, performing first-stage fine tuning: performing the first-stage fine tuning on the model by using multi-scale self-supervised contrast learning on a transformer substation sample image; 3, second-stage fine tuning: on a local defect sample image, performing second-stage fine tuning on the model by using supervised contrast learning; and 4, obtaining a model fine tuning result. According to the transformer substation defect detection model fine tuning method based on cascade comparative learning, smoother knowledge migration is realized, and the model performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of transformer substation defect detection, and in particular to a transformer substation defect detection model fine-tuning method based on cascade contrast learning. Background Art

[0002] The power system is a key infrastructure that affects the national economy, people's livelihood and national security. As the central link of "generation, transmission, transformation, distribution and use", the timely detection and treatment of substation defects is of great significance to maintaining the stable operation of the power grid. Affected by the natural environment, high-voltage loads and human factors, substations are prone to equipment defects such as foreign objects, breakage, oil leakage, and safety risks such as personnel smoking and not wearing safety helmets. For a long time, substation defects have relied on operation and maintenance personnel to discover through manual inspections. There are problems such as high labor intensity, low work efficiency, and many missed detections and false detections, which have brought hidden dangers to the safe operation of the power system. With the introduction of the concept of power vision and the development of the ubiquitous power Internet of Things, the automatic detection of substation defects based on computer vision technology and artificial intelligence technology has become a hot topic in research and application. That is, the surveillance cameras and inspection robots deployed in the substation are used to collect images of substation equipment, and then artificial intelligence algorithms are used to intelligently analyze and detect defects on the collected images. For example Figure 1 As shown, common substation defects include: broken meters, broken insulators, oil leakage, metal rust, broken respirators, abnormal door closure, oil stains on component surfaces, hanging suspended matter, bird nests, damaged doors, windows, walls and floors, broken covers, unlocked frame ladders, intrusion across the line, missing covers, not wearing safety helmets, not wearing work clothes, not wearing safety ropes, smoking, people falling to the ground, fire smoke, blurred dials, wall leakage, small animals intruding, abnormal meter readings, abnormal oil level of respirator oil seals, discoloration of silicone, water on the ground, roof leakage, etc.

[0003] The number of substation defect samples is scarce. In order to improve the performance of deep learning models on small sample sets, the transfer learning method is usually adopted. The model is first pre-trained on the ImageNet sample set, and then the parameters are fine-tuned on the substation defect sample set. However, due to the large difference in feature distribution between the samples in the ImageNet dataset and the substation defect samples, direct model fine-tuning is prone to negative transfer learning.

[0004] Substations generate a large number of unlabeled normal samples during daily inspections. Self-supervised learning methods represented by contrastive learning can make full use of these unlabeled normal samples to provide richer training samples for transfer learning. However, due to the lack of labeled prior information, contrastive learning is difficult to use non-homologous (i.e., from different samples) but similar sample information, resulting in similar instances being forced to separate in the feature space. In addition, contrastive learning can only perform feature learning based on the entire image, while substation defects usually only occupy a small part of the image. The learning model based on the entire image easily leads to model training biased towards the background area. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0006] To this end, the present invention proposes a substation defect detection model fine-tuning method based on cascade contrastive learning to achieve smoother knowledge transfer and improve model performance.

[0007] According to an embodiment of the present invention, a method for fine-tuning a substation defect detection model based on cascade contrastive learning comprises the following steps: Step 1, model pre-training: pre-training the model on an ImageNet sample image, that is, pre-training the ResNet50 backbone network on an ImageNet data set to obtain the initial weights of the ResNet50 backbone network; Step 2, first-level fine-tuning: on a substation sample image, first-level fine-tuning the model is performed using multi-scale self-supervised contrastive learning, that is, a self-supervised visual feature extraction framework based on SimCLR is adopted, data enhancement of input samples is used as positive samples, other inputs in the same batch are used as negative samples, and the feature distance between the input sample domain and the positive sample is maximized through a contrast loss function. At the same time, the feature distance between the negative samples in the input sample domain is minimized, and the sample features are learned accordingly; the self-supervised contrastive learning in this stage is performed on the entire substation sample, and the parameters of the ResNet50 backbone network are updated after training to complete the first stage of image-level contrastive learning, realizing the migration of natural image features to substation features; the third step, the second-level fine-tuning: on the local defect sample image, the model is fine-tuned at the second level using supervised contrastive learning, that is, label-guided SimCLR contrastive learning is used, and the inputs of the same category are all used as positive sample pairs. After training SimCLR with the constructed positive and negative sample pairs, the second stage of target-level contrastive learning is completed, realizing the migration of substation features to defect features; the fourth step, the model fine-tuning results are obtained.

[0008] The beneficial effect of the present invention is that, through two-stage cascade contrastive learning, smooth domain knowledge transfer and fusion can be achieved, which can not only retain the underlying feature extraction ability learned by the model in the pre-training stage, but also improve the model's understanding of high-level defect semantics; in the first stage of contrastive learning, a self-supervised contrastive learning method is used to achieve feature transfer from natural images to substation images, alleviating the negative transfer problem; in the second stage of contrastive learning, local defect samples are first generated using label information, and then contrastive learning is performed to achieve feature transfer from substation images to defect images, thereby improving the prominence of defect features.

[0009] According to one embodiment of the present invention, the ResNet50 backbone network is composed of five layers, namely Conv1, Conv2, Conv3, Conv4 and Conv5; Conv1 is composed of a convolutional layer, and the convolution kernel size of the convolutional layer is , the number of channels is , the step length is ; The Conv2 consists of a maximum pooling layer and three convolution groups, and the pooling kernel size of the maximum pooling layer is , the step length is ; Each convolution group includes three convolution layers, namely the first convolution layer, the second convolution layer and the third convolution layer; the convolution kernel size of the first convolution layer is , the number of channels is ; The convolution kernel size of the second convolutional layer is , the number of channels is ; The convolution kernel size of the third convolutional layer is , the number of channels is ; Conv3 is composed of four convolution groups, each of which includes three convolution layers, namely the fourth convolution layer, the fifth convolution layer and the sixth convolution layer; the convolution kernel size of the fourth convolution layer is , the number of channels is ; The convolution kernel size of the fifth convolutional layer is , the number of channels is ; The convolution kernel size of the sixth convolutional layer is , the number of channels is ; The Conv4 is composed of six convolution groups, each of which includes three convolution layers, namely the seventh convolution layer, the eighth convolution layer and the ninth convolution layer; the convolution kernel size of the seventh convolution layer is , the number of channels is ; The convolution kernel size of the eighth convolutional layer is , the number of channels is ; The convolution kernel size of the ninth convolutional layer is , the number of channels is ; The Conv5 is composed of three convolution groups, each of which includes three convolution layers, namely the tenth convolution layer, the eleventh convolution layer and the twelfth convolution layer; the convolution kernel size of the tenth convolution layer is , the number of channels is ; The convolution kernel size of the eleventh convolutional layer is , the number of channels is ; The convolution kernel size of the twelfth convolutional layer is , the number of channels is .

[0010] According to one embodiment of the present invention, the weights of Conv1 are always in a frozen state.

[0011] According to one embodiment of the present invention, the algorithm flow of SimCLR is as follows: Step 11, SimCLR first performs a Perform two different data augmentation operations to obtain this sample Two enhanced samples , , and , Considered as a positive sample pair; Step 12, backbone network from , Extract the representation vector and ; Step 13, and Then through the multi-layer perceptron Mapping to representation vector and .

[0012] According to one embodiment of the present invention, the SimCLR is maximized by the following loss function: and Similarity: (1) The meanings of the symbols in formula (1) are: Represents a positive sample pair , The loss function of Represents enhanced samples The corresponding representation vector; Represents enhanced samples The corresponding representation vector; Indicates The representation vector of the enhanced samples; express and The cosine similarity between ; express and The cosine similarity between ; represents the temperature parameter; represents the indicator function, and ;when When The value is 1; when When , the indicator function takes the value of 0; represents the number of samples, ,and is a positive integer; Represents the number of samples obtained after pairwise enhancement of a small batch of samples N.

[0013] According to one embodiment of the present invention, the SimCLR method has the following specific process when training each batch: Step 21: first randomly sample from the sample set Samples , for any sample Two different data enhancement operations are performed to obtain two enhanced samples and ; The samples finally obtained Enhanced samples , and for any , and are two enhanced samples from the same sample, i.e. They are two enhanced samples from the same sample; Step 22, for any enhanced sample , after being extracted by the backbone network, the output features of the Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer are obtained, respectively , , and ; Step 23, for any two enhanced samples and In the Output features of the layer and , calculated using formula (1) and The similarity loss of The calculation formula for the total similarity loss of all samples in the layer is: (2) The meanings of the symbols in formula (2) are: is the similarity loss value corresponding to the feature of scale m; It is an enhanced sample In the Output features of the layer; It is an enhanced sample In the Output features of the layer; Represents a positive sample pair The loss function of Represents a positive sample pair The loss function of .

[0014] According to one embodiment of the present invention, in the second step, similarity loss is calculated for the output features of the Conv2 layer, the Conv3 layer, the Conv4 layer and the Conv5 layer, and the total loss is obtained by weighted summation. The calculation formula is as follows: (3) The meanings of the symbols in formula (3) are: is the weight coefficient. The value range of m is 1, 2, 3 and 4, corresponding to Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv2 , the weight coefficient of Conv3 , the weight coefficient of Conv4 , the weight coefficient of Conv5 ; It is the similarity loss value corresponding to the feature of scale m.

[0015] According to one embodiment of the present invention, after the self-supervised contrastive learning is completed, the weights of the Conv2 and Conv3 layers are frozen.

[0016] According to one embodiment of the present invention, in the third step, the similarity loss is calculated for the output features of the Conv4 layer and the Conv5 layer, and the weighted sum is used to obtain the total loss. The calculation formula is as follows: (4) The meanings of the symbols in formula (4) are: is the weight coefficient. The value range of n is 1 and 2, corresponding to the Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv4 , the weight coefficient of Conv5 ; is the similarity loss value corresponding to the feature of scale n.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a schematic diagram of common substation defects; Figure 2 It is a smooth knowledge transfer graph from the natural domain to the substation domain and then to the defect domain realized by cascade contrastive learning; Figure 3 It is the structural diagram of the YOLOv5 model; Figure 4 It is the overall flow chart of cascade contrastive learning; Figure 5 It is the algorithm flow chart of SimCLR; Figure 6 It is a schematic diagram of multi-scale self-supervised contrastive learning; Figure 7 It is a schematic diagram of generating a local defect sample image from a substation sample image. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] A method for fine-tuning a substation defect detection model based on cascade contrast learning according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] See Figure 4 The substation defect detection model fine-tuning method based on cascade contrastive learning of the present invention comprises the following steps: Step 1: Model pre-training: Pre-train the model on ImageNet sample images, that is, pre-train the ResNet50 backbone network on the ImageNet dataset to obtain the initial weights of the ResNet50 backbone network.

[0024] The backbone network of the present invention adopts ResNet50, which consists of five layers: Conv1, Conv2, Conv3, Conv4 and Conv5. Among them, Conv1 consists of a convolution kernel size of , the number of channels is , the step length is Conv2 consists of a pooling kernel size of , the step length is The maximum pooling layer and three convolution groups are composed of three convolution layers, each of which has a convolution kernel size of , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is Conv3 consists of four convolution groups, each of which includes three convolution layers, namely: , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is Conv4 consists of six convolution groups, each of which includes three convolution layers, namely: , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is Conv5 consists of three convolution groups, each of which includes three convolution layers, namely: , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is , the convolution kernel size is , the number of channels is , the step value is .

[0025] That is to say, the ResNet50 backbone network of the present invention is composed of five layers, namely Conv1, Conv2, Conv3, Conv4 and Conv5; Conv1 is composed of one convolutional layer, and the convolution kernel size of the convolutional layer is , the number of channels is , the step length is ; Conv2 consists of a maximum pooling layer and three convolution groups. The pooling kernel size of the maximum pooling layer is , the step length is ; Each convolution group consists of three convolution layers, namely the first convolution layer, the second convolution layer and the third convolution layer; the convolution kernel size of the first convolution layer is , the number of channels is , the step value is ; The convolution kernel size of the second convolutional layer is , the number of channels is , the step value is ; The convolution kernel size of the third convolution layer is , the number of channels is , the step value is ; Conv3 consists of four convolution groups, each of which includes three convolution layers, namely the fourth convolution layer, the fifth convolution layer and the sixth convolution layer; the convolution kernel size of the fourth convolution layer is , the number of channels is , the step value is ; The convolution kernel size of the fifth convolutional layer is , the number of channels is , the step value is ; The convolution kernel size of the sixth convolutional layer is , the number of channels is , the step value is ; Conv4 consists of six convolution groups, each of which includes three convolution layers, namely the seventh convolution layer, the eighth convolution layer, and the ninth convolution layer; the convolution kernel size of the seventh convolution layer is , the number of channels is , the step value is ; The convolution kernel size of the eighth convolutional layer is , the number of channels is , the step value is ; The convolution kernel size of the ninth convolution layer is , the number of channels is , the step value is ; Conv5 consists of three convolution groups, each of which includes three convolution layers, namely the tenth convolution layer, the eleventh convolution layer, and the twelfth convolution layer; the convolution kernel size of the tenth convolution layer is , the number of channels is , the step value is ; The convolution kernel size of the eleventh convolution layer is , the number of channels is , the step value is ; The convolution kernel size of the twelfth convolutional layer is , the number of channels is , the step value is .

[0026] Step 2, first-level fine-tuning (self-supervised contrastive learning): On the substation sample image, multi-scale self-supervised contrastive learning is used to perform the first-level fine-tuning of the model. That is, self-supervised contrastive learning adopts a self-supervised visual feature extraction framework based on SimCLR, which uses the data enhancement of the input sample as the positive sample and other inputs of the same batch as the negative sample. It also maximizes the feature distance between the positive samples in the input sample domain and minimizes the feature distance between the negative samples in the input sample domain through the contrast loss function, and learns the sample features in this way. The self-supervised contrastive learning in this stage is performed on the entire substation sample. After training and updating, the parameters of the ResNet50 backbone network complete the first stage of image-level contrastive learning, realizing the migration of natural image features to substation features.

[0027] It should be noted that the main improvement of the first-level fine-tuning compared with the original SimCLR is multi-scale. The original SimCLR only used the output of the last layer of the network, while the present invention used 4 layers (Conv2, Conv3, Conv4 and Conv5) output features to comprehensively calculate the similarity loss value. The input image here is a sample image of the substation, which can be a normal image or an inspection image with defects (the defects account for a small proportion in the image).

[0028] like Figure 5 As shown, the algorithm flow of SimCLR is: Step 11: SimCLR first performs a Perform two different data augmentation operations to obtain this sample Two enhanced samples , , and , is considered as a positive sample pair. The data enhancement operations include cropping, rotation, color perturbation and random Gaussian blur.

[0029] Step 12: Backbone Network from , Extract the representation vector and ; Step 13 and Then through the multi-layer perceptron Mapping to representation vector and .

[0030] Among them, the representation vector and It is the feature extracted by the backbone network. For ResNet50, it is the feature map output by Conv5. In other words, and It is the feature directly output by the backbone network. This feature must first be used using a multi-layer perceptron Perform feature mapping and become and After that, the representation vector (i.e., feature) is needed for similarity calculation.

[0031] SimCLR maximizes the following loss function and Similarity: (1) The meanings of the symbols in formula (1) are: Represents a positive sample pair , The loss function of Represents enhanced samples The corresponding representation vector; Represents enhanced samples The corresponding representation vector; Indicates The representation vector of the enhanced samples; express and The cosine similarity between ; express and The cosine similarity between ; represents the temperature parameter; represents the indicator function, and ;when When The value is 1; when When , the indicator function takes the value of 0; represents the number of samples, ,and is a positive integer; Represents the number of samples obtained after pairwise enhancement of a small batch of samples N.

[0032] The numerator of the loss function is Represents the similarity between positive sample pairs, and the denominator Represents the similarity between negative sample pairs. Therefore, the higher the similarity between positive sample pairs, the lower the similarity between negative sample pairs, and the smaller the value of the loss function.

[0033] It should be noted that the numerator of formula (1) is and From the same sample, it is a positive sample pair. Represents similarity, so the numerator represents the similarity between positive sample pairs. The denominator of formula (1) is a batch of N samples. Each sample undergoes two different data enhancements, becoming two enhanced samples. Therefore, the denominator is the sum of 2N enhanced samples. The two enhanced samples from the same sample are positive sample pairs, while the two enhanced samples from different samples are negative sample pairs. The denominator has an indicator function ,when When =1, and When means Calculate the similarity with all other enhanced samples except itself. There are 2N enhanced samples in total. Excluding itself, there are 2N-1 enhanced samples left. One of these 2N-1 is a positive sample (that is, From the same sample ), the remaining 2N-2 are negative samples, so the denominator actually represents the similarity between negative sample pairs.

[0034] When SimCLR is training, N image samples are sent in each batch, and each image sample is enhanced twice to obtain 2 enhanced samples, so the total number of samples is equivalent to 2N samples. Among these 2N samples, there are only N positive sample pairs, that is, two samples from the same image are positive sample pairs, and samples from different images are negative sample pairs. In other words, SimCLR inputs N samples each time it is trained, and each sample is enhanced twice to obtain two enhanced samples, so the total number of samples is equivalent to 2N. For any sample For example, only Only then do they constitute a positive sample pair, and the 2N-2 enhanced samples obtained by enhancing the other N-1 samples are negative sample pairs.

[0035] Specifically, the SimCLR method follows the following process when training each batch: Step 21: First, randomly sample from the sample set Samples , for any sample Two different data enhancement operations are performed to obtain two enhanced samples and ; The samples finally obtained Enhanced samples , and for any , and are two enhanced samples from the same sample, i.e. They are two different enhanced samples from the same sample.

[0036] Step 22: For any enhanced sample , after being extracted by the backbone network, the output features of the Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer are obtained, respectively , , and .

[0037] Step 23: For any two enhanced samples and In the Output features of the layer and , calculated using formula (1) and The similarity loss of The calculation formula for the total similarity loss of all samples in the layer is: (2) The meanings of the symbols in formula (2) are: is the similarity loss value corresponding to the feature of scale m; It is an enhanced sample In the Output features of the layer; It is an enhanced sample In the Output features of the layer; Represents a positive sample pair The loss function of Represents a positive sample pair The loss function of .

[0038] It should be noted that the total number of samples is N, that is , where the kth sample Enhanced to get two samples and , these two samples pass through the backbone network to obtain m=1,2,3,4 layers of output and That is, when k=1, get and , and so on.

[0039] The original SimCLR only constrains the similarity of the output of the last layer (Conv5 layer) of the backbone network, while ResNet50 has a natural feature pyramid structure, that is, from Conv1 The width and height of each layer of Conv5 feature map are the width and height of the previous layer feature map The feature maps of each layer are stacked to form a pyramid structure with the width and height of each layer being half of the previous layer. For example, the feature map output by the Conv1 layer is , the feature map output by the Conv2 layer is , the feature map output by the Conv3 layer is , the feature map output by the Conv4 layer is , the feature map output by the Conv5 layer is .

[0040] In order to impose more fine-grained similarity constraints on the information of the intermediate layers (Conv2, Conv3 and Conv4), the present invention extends the framework of SimCLR and calculates the similarity loss for the features of each scale m , and then perform weighted summation to obtain the total loss (such as Figure 6 shown): In the second step, the similarity loss is calculated for the output features of the Conv2 layer, Conv3 layer, Conv4 layer, and Conv5 layer, and the total loss is obtained by weighted summation. The calculation formula is as follows: (3) The meanings of the symbols in formula (3) are: The value range of m is 1 to 4, corresponding to the Conv2, Conv3, Conv4 and Conv5 layers respectively (the weights of the Conv1 layer are frozen). is the weight coefficient, and its values ​​are: That is to say, is the weight coefficient. The value range of m is 1, 2, 3 and 4, corresponding to Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv2 , the weight coefficient of Conv3 , the weight coefficient of Conv4 , the weight coefficient of Conv5 ; It is the similarity loss value corresponding to the feature of scale m.

[0041] The self-supervised contrastive learning in this stage is performed on the entire substation sample (normal inspection sample or defect sample). After training and updating, the parameters of the backbone network complete the first stage of image-level contrastive learning, realizing the migration of natural image features to substation features. After this stage is completed, the weights of the Conv2 and Conv3 layers are frozen.

[0042] and Coming from the same sample, the principle of SimCLR is to make the samples from the same image as similar as possible after the backbone network extracts features (that is, to make the similarity between two enhanced samples from the same sample as high as possible); to make the samples from different images as different as possible after the backbone network extracts features (that is, to make the similarity between enhanced samples from different samples as low as possible). The whole process does not require image label information, so it is self-supervised learning. But also because SimCLR does not use the label information of the sample, if the two samples x and y come from different images, even if they are the same defect type, SimCLR will force their features to be dissimilar, and in fact, samples of the same defect type should be similar in features, so supervised contrast learning is proposed, which uses the label information of the sample, and considers all samples of the same type as positive sample pairs, and samples of different types as negative sample pairs.

[0043] Step 3, second-level fine-tuning (supervised contrastive learning): On the local defect sample images, supervised contrastive learning is used to perform second-level fine-tuning on the model, that is, label-guided SimCLR contrastive learning is adopted, and inputs of the same category are all used as positive sample pairs. After training SimCLR with the constructed positive and negative sample pairs, the second stage of target-level contrastive learning is completed, and the migration from substation features to defect features is realized.

[0044] It should be noted that the main improvement of the second-level fine-tuning compared with the original SimCLR is that it is supervised. The original SimCLR only considers the two samples obtained by enhancing the same sample as a positive sample pair. In this case, even two enhanced samples of the same category but from different samples will be regarded as negative sample pairs, and it is believed that enhanced samples of the same category must be similar in features. Therefore, a supervised approach is used here to regard the two enhanced samples from the same sample and the enhanced samples belonging to the same category as positive sample pairs. In addition, the input image here is a local image containing only defects that is cut out from the substation image.

[0045] The first stage of transfer learning is performed on the entire substation sample image, and defects usually account for a small proportion of the image, so the feature extraction of the backbone network may be biased towards non-defective areas (i.e., background). In order to solve the above problems, the present invention adopts the second stage of supervised contrastive learning to further enhance the backbone network's ability to extract defect features. The supervised contrastive learning here refers to label-guided SimCLR contrastive learning. The guiding role of labels is reflected in the following aspects: 1. In the substation defect sample set, according to the marked defect position box, the defect area image is taken out, and its size is scaled to be consistent with the input of the backbone network to obtain the local defect sample set; it should be noted that: the defect sample refers to the image containing defects collected by the inspection equipment, and the defect only occupies a small part of the image, and the local defect sample refers to the local image obtained by cutting out the defect area from the defect image; 2. Construct positive sample pairs and negative sample pairs: In the local defect sample set, images with consistent defect categories constitute positive sample pairs, and images with inconsistent defect categories constitute negative sample pairs.

[0046] Similar to self-supervised contrastive learning, the similarity loss is calculated for the output features of the Conv4 and Conv5 layers (the weights of Conv1, Conv2, and Conv3 are all frozen), and the weighted sum is used to obtain the total loss. The calculation formula is as follows: In supervised contrastive learning, the similarity loss is calculated for the output features of the Conv4 layer and the Conv5 layer, and the weighted sum is used to obtain the total loss. The calculation formula is as follows: (4) The meanings of the symbols in formula (4) are: is the weight coefficient. The value range of n is 1 and 2, corresponding to the Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv4 , the weight coefficient of Conv5 ; is the similarity loss value corresponding to the feature of scale n.

[0047] SimCLR only takes two enhanced samples of the same image as positive sample pairs, which cannot effectively learn similar features of the same type of defects. Here, all inputs of the same category are taken as positive sample pairs to improve the network's generalization ability for the same type of defects. After training SimCLR with the constructed positive and negative sample pairs, the second stage of target-level contrastive learning is completed, realizing the migration of substation features to defect features.

[0048] Step 4: Get the model fine-tuning results.

[0049] In order to achieve smoother knowledge transfer and improve model performance, the present invention proposes a two-stage cascade contrastive learning method, such as Figure 2 As shown. In the first stage of contrastive learning, based on self-supervised contrastive learning (i.e., image-level contrastive learning), the migration from the natural domain to the substation domain is realized, that is, the image-level migration of natural image features to substation image features is realized, which alleviates the negative migration problem. In the second stage of contrastive learning, based on supervised contrastive learning (i.e., target-level contrastive learning), local defect samples are first generated using label information, and then contrastive learning is performed to realize the migration from the substation domain to the defect domain, that is, the target-level migration of substation image features to defect image features is realized, and the prominence of defect features is improved. Through the two-stage smooth transfer learning method, the underlying feature extraction capability learned by the model in the pre-training stage can be retained, and the model's understanding of high-level defect semantics can be improved.

[0050] The present invention adopts the YOLOv5 model to detect substation defects. The YOLOv5 model structure is as follows: Figure 3 As shown in Figure 2, the main process includes extracting feature maps from the input image through the backbone network, and then dividing the feature maps into grids (each scale The network is used to classify and regress each grid to obtain the detection result. The scale refers to the multi-scale feature map extraction, that is, the backbone network outputs feature maps of different sizes, and the feature maps of each scale are divided into grids, and perform classification and regression on each grid. It is a grid partitioning hyperparameter, and its value is usually 7.

[0051] Aiming at the problem of fine-tuning the model weights of the backbone network, the present invention proposes a method based on cascade contrastive learning, such as Figure 4 First, the backbone network is pre-trained on the ImageNet dataset to obtain the initial weights of the backbone network. The present invention uses the pre-trained weights `ResNet50_Weights.IMAGENET1K_V2` provided by PyTorch.

[0052] Since the Conv1 layer is mainly used to extract low-level features such as edges and textures, it has good task versatility (i.e., substation defect detection also needs to extract low-level features such as edges and textures first), therefore, in the present invention, the weight of Conv1 is always in a frozen state. After the self-supervised contrastive learning is completed, the weights of the Conv2 and Conv3 layers are frozen.

[0053] Since the image features in ImageNet are quite different from those in the substation defect sample set, the backbone network is prone to negative transfer when fine-tuning the weights on the substation defect sample set directly based on the pre-trained weights on ImageNet, resulting in a decrease in model performance. The present invention uses cascade contrastive learning, i.e., self-supervised contrastive learning and supervised contrastive learning, to achieve smooth backbone network knowledge transfer, such as Figure 4 shown.

[0054] Input of the SimCLR backbone network in self-supervised contrastive learning and is a single substation sample image The results obtained by two different data augmentations; in supervised contrastive learning, the input of the SimCLR backbone network and It is a local image containing only the defect area extracted from the defect sample, and no data enhancement operation is performed, such as Figure 7 As shown. Using local defect images to construct sample pairs can alleviate the problem of bias towards the background area when the entire image is used as input. In addition, a positive sample pair in self-supervised contrastive learning refers to two enhanced samples from the same image, and two enhanced samples from different images are considered negative sample pairs. Therefore, even if two enhanced samples come from the same type of defects, they will be considered negative sample pairs (this is because native SimCLR is a self-supervised algorithm and cannot use the label information of the samples). This will cause similar instances to be forced to separate in the feature space. In supervised contrastive learning, images of the same defect type are considered positive sample pairs, and images of different defect types are considered negative sample pairs, which can achieve more accurate feature similarity constraints.

[0055] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A substation defect detection model fine-tuning method based on cascade contrastive learning, characterized in that: The following steps are involved: Step 1: Model pre-training: Pre-train the model on ImageNet sample images, that is, pre-train the ResNet50 backbone network on the ImageNet dataset to obtain the initial weights of the ResNet50 backbone network; Step 2, first-level fine-tuning: On the substation sample image, multi-scale self-supervised contrastive learning is used to perform the first-level fine-tuning of the model, that is, a self-supervised visual feature extraction framework based on SimCLR is adopted, the data enhancement of the input sample is used as the positive sample, and other inputs in the same batch are used as the negative sample. The feature distance between the positive samples in the input sample domain is maximized through the contrast loss function, while the feature distance between the negative samples in the input sample domain is minimized, and the sample features are learned in this way; the self-supervised contrastive learning in this stage is performed on the entire substation sample, and the parameters of the ResNet50 backbone network are updated after training to complete the first stage of image-level contrastive learning, realizing the migration of natural image features to substation features; Step 3, second-level fine-tuning: On the local defect sample images, supervised contrastive learning is used to perform second-level fine-tuning on the model, that is, label-guided SimCLR contrastive learning is adopted, and inputs of the same category are all used as positive sample pairs. After training SimCLR with the constructed positive sample pairs and negative sample pairs, the second-stage target-level contrastive learning is completed, and the migration from substation features to defect features is realized; Step 4: Get the model fine-tuning results.

2. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 1 is characterized in that: The ResNet50 backbone network consists of five layers, namely Conv1, Conv2, Conv3, Conv4 and Conv5; The Conv1 consists of a convolutional layer, and the convolution kernel size of the convolutional layer is , the number of channels is , the step length is ; The Conv2 consists of a maximum pooling layer and three convolution groups. The pooling kernel size of the maximum pooling layer is , the step length is ; Each convolution group includes three convolution layers, namely the first convolution layer, the second convolution layer and the third convolution layer; the convolution kernel size of the first convolution layer is , the number of channels is ; The convolution kernel size of the second convolutional layer is , the number of channels is ; The convolution kernel size of the third convolutional layer is , the number of channels is ; Conv3 is composed of four convolution groups, each of which includes three convolution layers, namely the fourth convolution layer, the fifth convolution layer and the sixth convolution layer; the convolution kernel size of the fourth convolution layer is , the number of channels is ; The convolution kernel size of the fifth convolutional layer is , the number of channels is ; The convolution kernel size of the sixth convolutional layer is , the number of channels is ; Conv4 is composed of six convolution groups, each of which includes three convolution layers, namely the seventh convolution layer, the eighth convolution layer and the ninth convolution layer; the convolution kernel size of the seventh convolution layer is , the number of channels is ; The convolution kernel size of the eighth convolutional layer is , the number of channels is ; The convolution kernel size of the ninth convolutional layer is , the number of channels is ; Conv5 is composed of three convolution groups, each of which includes three convolution layers, namely the tenth convolution layer, the eleventh convolution layer and the twelfth convolution layer; the convolution kernel size of the tenth convolution layer is , the number of channels is ; The convolution kernel size of the eleventh convolutional layer is , the number of channels is ; The convolution kernel size of the twelfth convolutional layer is , the number of channels is .

3. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 2 is characterized in that: The weights of Conv1 are always frozen.

4. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 1 is characterized in that: The algorithm flow of the SimCLR is: Step 11: SimCLR first performs a Perform two different data augmentation operations to obtain this sample Two enhanced samples , , and , Considered as a positive sample pair; Step 12: Backbone Network from , Extract the representation vector and ; Step 13 and Then through the multi-layer perceptron Mapping to representation vector and .

5. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 4 is characterized in that: The SimCLR is maximized by the following loss function and Similarity: （1） The meanings of the symbols in formula (1) are: Represents a positive sample pair , The loss function of Represents enhanced samples The corresponding representation vector; Represents enhanced samples The corresponding representation vector; Indicates The representation vector of the enhanced samples; express and The cosine similarity between ; express and The cosine similarity between ; represents the temperature parameter; represents the indicator function, and ;when When The value is 1; when When , the indicator function takes the value of 0; represents the number of samples, ,and is a positive integer; Represents the number of samples obtained after pairwise enhancement of a small batch of samples N.

6. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 5 is characterized in that: The specific process of the SimCLR method when training each batch is as follows: Step 21: First, randomly sample from the sample set Samples , for any sample Two different data enhancement operations are performed to obtain two enhanced samples and ; The samples finally obtained Enhanced samples , and for any , and are two enhanced samples from the same sample, i.e. They are two enhanced samples from the same sample; Step 22: For any enhanced sample , after being extracted by the backbone network, the output features of the Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer are obtained, respectively , , and ; Step 23: For any two enhanced samples and In the Output features of the layer and , calculated using formula (1) and The similarity loss of The calculation formula for the total similarity loss of all samples in the layer is: （2） The meanings of the symbols in formula (2) are: is the similarity loss value corresponding to the feature of scale m; It is an enhanced sample In the Output features of the layer; It is an enhanced sample In the Output features of the layer; Represents a positive sample pair The loss function of Represents a positive sample pair The loss function of .

7. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 6 is characterized in that: In the second step, similarity loss is calculated for the output features of Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer, and the total loss is obtained by weighted summation. The calculation formula is as follows: （3） The meanings of the symbols in formula (3) are: is the weight coefficient. The value range of m is 1, 2, 3 and 4, corresponding to Conv2 layer, Conv3 layer, Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv2 , the weight coefficient of Conv3 , the weight coefficient of Conv4 , the weight coefficient of Conv5 ; It is the similarity loss value corresponding to the feature of scale m.

8. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 7 is characterized in that: When self-supervised contrastive learning is completed, freeze the weights of Conv2 and Conv3 layers.

9. The method for fine-tuning a substation defect detection model based on cascade contrastive learning according to claim 8, characterized in that: In the third step, the similarity loss is calculated for the output features of the Conv4 layer and the Conv5 layer, and the total loss is obtained by weighted summation. The calculation formula is as follows: （4） The meanings of the symbols in formula (4) are: is the weight coefficient. The value range of n is 1 and 2, corresponding to the Conv4 layer and Conv5 layer respectively. The weight coefficient of Conv4 , the weight coefficient of Conv5 ; is the similarity loss value corresponding to the feature of scale n.

Images