A device defect detection method under few sample conditions
By combining a self-supervised contrastive learning model with a few-shot classification model and training a feature extractor using an unlabeled dataset, the problem of scarce labeled data in power transmission equipment defect detection is solved, and the model's recognition accuracy and generalization ability are improved.
Patent Information
- Application Number
- CN202411154835.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-22
AI Technical Summary
In computer vision inspection of defects in power transmission equipment, the existing methods cannot build high-quality labeled training sets to meet the contradiction between the need for high precision and the scarcity of labeled data. This results in low model recognition accuracy, and data augmentation methods may lead to model overfitting and insufficient generalization ability.
We combine a self-supervised contrastive learning model with a few-shot classification model. We train a feature extractor on an unlabeled dataset using self-supervised contrastive learning and extract features by utilizing the differences between the foreground and background in the image. This constructs a multi-task detection model and reduces the reliance on labeled samples.
It improves the prediction accuracy for fewer sample classes, avoids model overfitting, enhances generalization ability, reduces dependence on labeled samples, and achieves efficient equipment defect detection.
Smart Images

Figure CN118657786B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart grid technology, and in particular to a method for detecting equipment defects under a small number of samples. Background Art
[0002] The normal operation of power transmission equipment is the basis for the stability of the power system. When equipment has defects, timely identification and replacement can avoid systemic failures. However, in the field of computer vision detection of power transmission equipment defects, there is a contradiction between high-precision requirements and the scarcity of labeled data. There are often few image samples of labeled defective equipment. If a supervised training model is used, the training set depends on a large number of data samples and manual annotations. For the few-sample category, it is impossible to build a high-quality labeled training set, resulting in low final recognition accuracy of the model. In addition, the existing few-sample classification model uses data enhancement methods to expand data samples. Since the expanded samples are similar to the original data features, the model may overfit to the training data and lack generalization ability, resulting in poor final recognition accuracy of the model. Summary of the invention
[0003] The purpose of the present invention is to improve and innovate the shortcomings and problems existing in the background technology and to provide a device defect detection method under a small number of samples.
[0004] According to a first aspect of the present invention, a method for detecting device defects under a few-sample condition is provided, which specifically comprises the following steps:
[0005] Construct an image dataset D, and divide the image dataset D into a few-sample classification labeled dataset D1 and a self-supervised contrastive learning unlabeled dataset D2;
[0006] Construct and train a multi-task detection model; wherein the multi-task detection model includes a self-supervised contrastive learning model and a few-shot classification model, and the self-supervised contrastive learning model and the few-shot classification model share a feature extractor; the self-supervised contrastive learning model uses a self-supervised contrastive learning unlabeled data set D2 to complete feature extractor training, and the few-shot classification model uses the trained feature extractor to extract features from the few-shot classification labeled data set D1, and uses the extracted image feature map to complete the training of the embedding module in the few-shot classification model and fine-tune the feature extractor;
[0007] The trained multi-task detection model is used to detect images of power transmission equipment.
[0008] A further solution is that the few-sample classification annotation dataset D1 includes a support set and a query set, and the image samples in the support set and the query set share the same category.
[0009] A further solution is that the self-supervised contrastive learning model uses the self-supervised contrastive learning unlabeled data set D2 to complete the feature extractor training, specifically including:
[0010] The image samples belonging to the self-supervised contrastive learning unlabeled dataset D2 in the same training batch are input into the pre-trained MASK R-CNN network; the MASK R-CNN network uses the backbone network to extract the image feature map of each image sample;
[0011] Performing an activation operation on the image feature map of the image sample to obtain a foreground activation map and a background activation map of the image sample;
[0012] Based on the foreground activation map, background activation map and image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation ;
[0013] Any two foreground feature representations or two background feature representations constitute a positive pair, and one foreground feature representation and one background feature representation constitute a negative pair; the positive pairs are sorted by feature similarity, and weights are assigned to the positive pairs based on the sorting order value to reduce the interference of low-similarity positive pairs;
[0014] Design a contrastive learning loss function to pull the representations of positive pairs closer together and pull the representations of negative pairs further apart;
[0015] Calculate the contrastive learning loss and back-propagate to update the parameters of the MASK R-CNN network.
[0016] A further solution is that the activation operation is performed on the image feature map of the image sample to obtain the foreground activation map and the background activation map of the image sample specifically includes:
[0017] The MASK R-CNN network is used to perform semantic segmentation on the image feature map to obtain the segmentation result to distinguish the foreground area and background area of the image sample;
[0018] Map the foreground area and background area of the segmentation result back to the image feature map to distinguish the feature points belonging to the foreground area and the background area in the image feature map;
[0019] The feature points of the foreground area are retained, the feature points of the background area are removed, and the channel dimension reduction and normalization operations are performed on the image feature map of the retained foreground area to obtain the foreground activation map of the image sample; at the same time, the feature points of the background area are retained, the feature points of the foreground area are removed, and the channel dimension reduction and normalization operations are performed on the image feature map of the retained background area to obtain the background activation map of the image sample.
[0020] A further solution is that, based on the foreground activation map, the background activation map and the image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation Specifically include:
[0021] Flatten the foreground activation map and the background activation map into a one-dimensional vector; Flatten the image feature map extracted by the backbone network of the MASK R-CNN network into a single-channel matrix;
[0022] Based on the tiled foreground activation map, background activation map and image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation Among them, Foreground feature representation of samples and background feature representation It can be expressed as:
[0023] ;
[0024] In the formula, , , They represent the foreground activation map, background activation map, and image feature map after tiling respectively; and , , .
[0025] A further solution is that the method of sorting the positive pairs by using feature similarity and assigning weights to the positive pairs based on the sorting order value to reduce the interference of the positive pairs with low similarity specifically includes:
[0026] Calculate the cosine similarity of each pair, and then sort the cosine similarities of each foreground pair and background pair respectively, assign large weights to pairs with high similarity, and assign small weights to pairs with low similarity. The weight assignment formula for pairs is as follows:
[0027] ;
[0028] represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where A hyperparameter for controlling the smoothness of the exponential function, and Indicates the corresponding sort order value.
[0029] A further solution is that the contrast loss function is designed as:
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037] ;
[0038] represents the loss value of the prospect, and Represents the loss value of the background. Represents the overall positive loss value, represents the loss value of the negative pair, represents the overall loss value; where n represents the number of image samples belonging to the unlabeled dataset D2 of the same training batch; represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where yes and The cosine similarity between ; yes and The cosine similarity between yes and The cosine similarity between .
[0039] A further solution is that the few-sample classification model uses the trained feature extractor to extract features from the few-sample classification annotated dataset D1, and uses the extracted image feature map to complete the training of the embedding module in the few-sample classification model and the fine-tuning of the feature extractor. Specifically, it includes:
[0040] Input the image samples belonging to the few-sample classification annotation dataset D1 in the same training batch into the trained MASK R-CNN network; the MASK R-CNN network uses the backbone network to extract the image feature map of the corresponding image sample;
[0041] The image feature maps corresponding to the support set and the query set are input into the embedding module, which is used to convert the high-dimensional feature matrix into a low-dimensional feature vector;
[0042] Calculate the average value of feature vectors of multiple image samples of the same category in the support set, and calculate the similarity between the feature vectors of the query set image samples and the average value of feature vectors corresponding to each category in the support set through the distance measurement module;
[0043] The calculated similarity is assigned to the softmax function to obtain the probability distribution of the query set image samples;
[0044] Based on the loss between the predicted and true values of the query set image samples, backpropagation updates the embedding module parameters and fine-tunes the feature extractor parameters.
[0045] According to a second aspect of the present invention, there is provided an electronic device comprising a processor and a memory, wherein the processor is configured to implement the steps of a method for detecting device defects under a small number of samples as described in any one of the above items when executing a computer program stored in the memory.
[0046] According to a third aspect of the present invention, there is provided a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of a method for detecting device defects under a few sample conditions as described in any one of the above items are implemented.
[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a method for detecting equipment defects under the condition of few samples, and uses self-supervised contrastive learning to train a common feature extractor with an unlabeled data set D2. Compared with the supervised training mode, which requires a large number of labeled samples to train the feature extractor, the present invention reduces the dependence on labeled samples during training and improves the prediction accuracy of the few-sample category. Moreover, compared with the conventional few-sample classification model that expands the labeled samples by data enhancement, the training effect is not good because the expanded samples are similar to the original data features; the present invention uses a self-supervised contrastive learning model to train the feature extractor, and can use a large number of unlabeled data sets D2 to train the feature extractor. On the one hand, it reduces the dependence on labeled samples, and can train an effective feature extractor using only unlabeled data, which significantly reduces the dependence on labeled samples; on the other hand, it also avoids the problem that the model may overfit to the training data and the generalization ability is insufficient due to the similarity of the expanded samples to the original data features, resulting in poor training effect. The self-supervised contrastive learning model of the present invention uses the difference between the foreground and background in the image as a contrast, constructs a self-supervised contrastive learning model based on the foreground and background, learns feature representation from a large amount of unlabeled data to train the feature extractor, and avoids the limitation of the similarity of the original data features. The present invention integrates multiple models, reduces the dependence on labeled samples during training, and improves the prediction accuracy of small sample categories. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 It is a schematic diagram of a device defect detection method under a small number of sample conditions provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0052] See also Figure 1 The present invention provides a device defect detection method under a few sample conditions, which specifically includes the following steps:
[0053] Step S1, construct an image dataset D, and divide the image dataset D into a few-sample classification labeled dataset D1 and a self-supervised contrastive learning unlabeled dataset D2;
[0054] In this embodiment, multiple categories of power transmission equipment images are obtained in advance to construct an image dataset; the image dataset D is divided into a few-sample classification labeled dataset D1 and a self-supervised contrastive learning unlabeled dataset D2, and the few-sample classification labeled dataset D1 is further divided into a support set and queryset .
[0055] It should be noted that the present invention is to detect whether the power transmission equipment has defects, so as to identify and replace the defective power transmission equipment in time and ensure the stable operation of the power system. However, there are often only a small number of labeled samples of defective power transmission equipment. The present invention constructs a small number of labeled defective power transmission equipment image samples into a small number of sample classification and labeling data set D1; and a large number of unlabeled power transmission equipment image samples are used to construct a self-supervised comparative learning unlabeled data set D2. The defective power transmission equipment can be a cracked lightning arrester, a broken insulator, an offset shock-absorbing hammer, etc.; wherein the categories of the image samples in the small number of sample classification and labeling data set D1 and the self-supervised comparative learning unlabeled data set D2 do not intersect with each other; for example, when the small number of sample classification and labeling data set D1 includes image samples of cracked lightning arresters, broken insulators and offset shock-absorbing hammers, the self-supervised comparative learning unlabeled data set D2 does not include image samples of cracked lightning arresters, broken insulators and offset shock-absorbing hammers. In addition, cracked arresters and normal arresters belong to different categories; although the image samples are of different categories, the foreground appearance of the image samples is similar, and the background color or texture of the image samples is similar. The self-supervised contrastive learning unlabeled dataset D2 can include normal insulator and shock-absorbing hammer image samples.
[0056] It should be further explained that the present invention further divides the few-sample classification annotation dataset D1 into the support set and queryset ; Among them, the support set and queryset The image samples in share the same category. There are M categories, each with K samples; the query set M categories are also included.
[0057] Step S2, constructing and training a multi-task detection model; wherein the multi-task detection model includes a self-supervised contrastive learning model and a few-sample classification model, and the self-supervised contrastive learning model and the few-sample classification model share a feature extractor; the self-supervised contrastive learning model uses the self-supervised contrastive learning unlabeled data set D2 to complete feature extractor training, and the few-sample classification model uses the trained feature extractor to extract features from the few-sample classification labeled data set D1, and uses the extracted image feature map to complete the training of the embedding module in the few-sample classification model and the fine-tuning of the feature extractor;
[0058] Specifically, a multi-task detection model is constructed; wherein the multi-task detection model includes a self-supervised contrastive learning model and a few-shot classification model, and the self-supervised contrastive learning model and the few-shot classification model share a feature extractor. The architecture of the multi-task detection model is hard parameter sharing, and the overall model parameters are divided into shared hard parameters and sub-task unique parameters. The shared hard parameters refer to the parameters of the feature extractor.
[0059] In this embodiment, the feature extractor uses a MASK R-CNN network, which uses a backbone network (such as ResMet) to extract image features, and the extracted image feature map is used for target detection and segmentation tasks. The MASK R-CNN network allows accurate pixel-level segmentation of the target image. It should be noted that the MASK R-CNN network of the present invention is pre-trained on imageMet1K and already has certain semantic segmentation capabilities.
[0060] Among them, the self-supervised contrastive learning model uses the difference between the foreground and background in the image sample as a contrast to construct a self-supervised contrastive learning model based on the foreground and background; the process of training the feature extractor of the self-supervised contrastive learning model is as follows:
[0061] (1) Input the image samples belonging to the self-supervised contrastive learning unlabeled dataset D2 in the same training batch into the pre-trained Mask R-CNN network; the Mask R-CNN network uses the backbone network to extract the image feature map of each image sample;
[0062] During the training process, a large number of image samples are usually divided into multiple batches for processing. The image samples in each batch are input into the model together, and the loss is calculated and the model parameters are updated through forward propagation and back propagation. The size of the batch can be set according to demand. In this embodiment, the same training batch includes both image samples in the self-supervised contrastive learning unlabeled dataset D2 and image samples in the few-sample classification labeled dataset D1.
[0063] (2) The Mask R-CNN network further performs semantic segmentation on the image feature map to obtain a segmentation result to distinguish the foreground and background areas of the image sample;
[0064] (3) Mapping the foreground area and background area of the segmentation result back to the image feature map to distinguish the feature points belonging to the foreground area and the background area in the image feature map;
[0065] (4) retaining the feature points of the foreground area, removing the feature points of the background area, and performing channel dimensionality reduction and normalization operations on the image feature map of the retained foreground area to obtain the foreground activation map of the image sample; at the same time, retaining the feature points of the background area, removing the feature points of the foreground area, and performing channel dimensionality reduction and normalization operations on the image feature map of the retained background area to obtain the background activation map of the image sample;
[0066] For example, for the i-th image sample, the MASK R-CNN network uses the backbone network to extract the image feature map of the i-th image sample , and map the foreground and background areas of the segmentation results back to the image feature map In order to distinguish the image feature map The feature points belonging to the foreground area and the background area are obtained; then, the feature points of the foreground area are retained, the feature values of the feature points corresponding to the background area are assigned to zero, and the image feature map of the retained foreground area is subjected to channel dimensionality reduction, the feature values of each channel of the feature points are summed, and the image feature map is converted into a single-channel image feature map, and the size of the image feature map remains unchanged; then, the single-channel image feature map is normalized so that the feature values of each feature point corresponding to the foreground area are between 0 and 1, so as to obtain the foreground activation map of the image sample. ; Similarly, the same method is used to obtain the background activation map of the image sample.
[0067] (5) Flatten the foreground activation map and the background activation map into a one-dimensional vector; Flatten the image feature map extracted by the backbone network of the MASK R-CNN network into a single-channel matrix;
[0068] For example, for the foreground activation map , tile the foreground activation map into One-dimensional vector; for the image feature map extracted by the backbone network , flatten the image feature map into A single channel matrix.
[0069] (6) Based on the tiled foreground activation map, background activation map and image feature map, the image feature map of the image sample is transformed into Decomposed into foreground feature representation and background feature representation Among them, Foreground feature representation of samples and background feature representation It can be expressed as:
[0070] ;
[0071] In the formula, , , They represent the foreground activation map, background activation map, and image feature map after tiling respectively; and , , .
[0072] (7) Any two foreground feature representations or two background feature representations constitute a positive pair, and one foreground feature representation and one background feature representation constitute a negative pair; the positive pairs are sorted by feature similarity, and weights are assigned to the positive pairs based on the sorting order value to reduce the interference of low-similarity positive pairs;
[0073] Specifically, we first calculate the cosine similarity of each pair, and then sort the cosine similarities of each foreground pair and background pair, assigning large weights to pairs with high similarity and small weights to pairs with low similarity. The weight assignment formula for pairs is as follows:
[0074] ;
[0075] represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where A hyperparameter for controlling the smoothness of the exponential function, and Represents the corresponding sorting order value of the foreground pair. If the same training batch belongs to the self-supervised contrastive learning unlabeled dataset D2 with n image samples, then for the foreground pair there are Yes; there is also one in the background Yes, the sort order values are from 1 to .
[0076] (8) Design a contrastive learning loss function to pull the representations of positive pairs closer together and pull the representations of negative pairs further apart. The contrastive loss function is designed as:
[0077]
[0078]
[0079]
[0080]
[0081]
[0082]
[0083]
[0084] ;
[0085] represents the loss value of the prospect, and Represents the loss value of the background. Represents the overall positive loss value, represents the loss value of the negative pair, represents the overall loss value; where n represents the number of image samples belonging to the unlabeled dataset D2 of the same training batch; represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where yes and The cosine similarity between ; yes and The cosine similarity between yes and The cosine similarity between .
[0086] (9) Calculate the contrastive learning loss and back-propagate to update the parameters of the MASK R-CNN network.
[0087] Specifically, the process of training the embedding module of the few-shot classification model is as follows:
[0088] (1) Input the image samples belonging to the few-shot classification and annotation dataset D1 in the same training batch into the trained Mask R-CNN network; the Mask R-CNN network uses the backbone network to extract the image feature map of the corresponding image sample;
[0089] Since the self-supervised contrastive learning model and the few-shot classification model share a feature extractor, during the training of the few-shot classification model, the feature extractor trained by the self-supervised contrastive learning model is used to process the image samples belonging to the few-shot classification labeling dataset D1 in the same training batch to extract the corresponding image feature map.
[0090] It should be noted that the image samples belonging to the few-sample classification annotation dataset D1 in the same training batch include a support set and a query set, and the support set and the query set share the same category. Therefore, the image feature map of the corresponding support set image sample and the image feature map of the query set image sample will be extracted.
[0091] (2) Input the image feature maps corresponding to the support set and the query set into the embedding module, which is used to convert the high-dimensional feature matrix into a low-dimensional feature vector;
[0092] The embedding module can be an existing CNN and VGG neural network model. The embedding module is used to convert the image feature map into a low-dimensional feature vector so that the distance measurement module can calculate the Euclidean distance between image samples.
[0093] (3) Calculate the average value of the feature vectors of multiple image samples of the same category in the support set, and calculate the similarity between the feature vectors of the query set image samples and the average value of the feature vectors corresponding to each category in the support set through the distance measurement module;
[0094] As mentioned above, the support set includes M categories. For the same training batch, the average values of the feature vectors of multiple image samples belonging to the same category are obtained, so that the average values of M feature vectors can be calculated; wherein, the distance measurement module calculates the similarity between the feature vector of the query set image sample and the average value of the M feature vectors, and the Euclidean distance between the feature vector of the query set image sample and the average value of the M feature vectors can be calculated.
[0095] (4) The similarity between the feature vector of the query set image sample and the average value of the feature vector corresponding to each category in the support set is assigned to the softmax function to obtain the probability distribution of the query set image sample;
[0096] Specifically, by assigning the Euclidean distance between the feature vector of the image sample and the average value of the feature vector of each category to the softmax function, the probability distribution of the query set image samples can be output, and the prediction value of the query set image samples belonging to each category can be obtained.
[0097] (5) Based on the loss between the predicted value and the true value of the query set image sample, back-propagation updates the embedding module parameters and fine-tunes the feature extractor parameters;
[0098] Specifically, design the classification loss function: , where M represents the total number of image sample categories in the support set, represents the predicted value of the query set image sample, Represents the true value of the query set image sample. After the Euclidean distance between the feature vector of the query set image sample and the average value of the feature vector of each category is assigned to the softmax function, the predicted value of the image sample in each category can be obtained. Since the image samples of the query set are also labeled data; therefore, the loss between the predicted value and the true value of the query set image sample can be calculated through the classification loss function, and the feature extractor and embedding module parameters can be back-propagated to update.
[0099] It should be noted that the multi-task detection model of the present invention includes a self-supervised contrastive learning model and a few-sample classification model, and the self-supervised contrastive learning model and the few-sample classification model share a feature extractor. The architecture of the multi-task detection model is hard parameter sharing, and the overall model parameters are divided into shared parameters and sub-task unique parameters. The shared parameters refer to the parameters of the feature extractor. Therefore, based on the loss between the predicted value and the true value of the query set image sample, the parameters of the feature extractor will also be back-propagated and updated.
[0100] Both the self-supervised contrastive learning model and the few-shot classification model use the backbone network of the Mask R-CNN network as a feature extractor. The self-supervised contrastive learning model also uses the Mask R-CNN network for semantic segmentation. The few-shot classification model also contains the unique parameters of the backbone network. Therefore, in order to balance the loss functions of the two subtasks of self-supervised contrastive learning and few-shot classification, the overall model objective loss function is:
[0101] ;
[0102] In the formula, λ and μ are used to measure the weights of the few-shot classification task and the contrastive learning task, respectively. The self-supervised contrastive learning unlabeled dataset D2 is used to train the self-supervised contrastive learning model, and the parameters of the MASK R-CNN network are updated. The few-shot classification labeled dataset D1 is used to train the few-shot classification model, and the parameters of the embedding module and the backbone network of the MASK R-CNN network are updated until the overall model objective loss function converges.
[0103] Step S3: Using the trained multi-task detection model to detect the image of the power transmission equipment;
[0104] It should be noted that after the multi-task detection model is trained, it can be used to detect defects in power transmission equipment; at this time, the image samples input to the multi-task detection model also include the support set and the query set. The image samples to be detected belong to the query set. At this time, the image samples of the query set are unlabeled image samples; and the image samples of the support set are image samples with labels of multiple categories. After the image feature map of the image sample is extracted by a common feature extractor, the high-dimensional image feature map is converted into a low-dimensional feature vector through an embedding module; based on the Euclidean distance between the feature vector of the query set and the average value of the feature vector of each category in the support set, the category to which the image sample to be detected belongs can be accurately identified.
[0105] In summary, the present invention provides a device defect detection method under few-sample conditions, which uses self-supervised contrast learning to train a common feature extractor using an unlabeled data set D2. Compared with the supervised training mode, which requires a large number of labeled samples to train the feature extractor, the present invention reduces the dependence on labeled samples during training and improves the prediction accuracy of the few-sample category. Moreover, compared with the conventional few-sample classification model, which expands the labeled samples by data enhancement, the training effect is poor due to the similarity between the expanded samples and the original data features; the present invention uses a self-supervised contrast learning model to train the feature extractor, and can use a large number of unlabeled data sets D2 to train the feature extractor. On the one hand, it reduces the dependence on labeled samples, and only uses unlabeled data to train an effective feature extractor, which significantly reduces the dependence on labeled samples; on the other hand, it also avoids the problem that the model may be overfitted to the training data and the generalization ability is insufficient due to the similarity between the expanded samples and the original data features, resulting in poor training results. The self-supervised contrast learning model of the present invention uses the difference between the foreground and the background in the image as a comparison, constructs a self-supervised contrast learning model based on the foreground and the background, learns feature representations from a large amount of unlabeled data to train the feature extractor, and avoids the limitation of the similarity of the original data features. The present invention integrates multiple models, reduces the dependence on labeled samples during training, and improves the prediction accuracy of small sample categories.
[0106] The present invention provides an electronic device, comprising a processor and a memory, wherein the processor is used to implement the steps of a device defect detection method under a few sample conditions as described in the above embodiment when executing a computer program stored in the memory.
[0107] The present invention provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a device defect detection method under a few sample conditions as described in the above embodiment are implemented.
[0108] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the technical field without departing from the purpose of the present invention.
Claims
1. A device defect detection method under few sample conditions, characterized in that: The specific steps include: Construct an image dataset D, and divide the image dataset D into a few-sample classification labeled dataset D1 and a self-supervised contrastive learning unlabeled dataset D2; Construct and train a multi-task detection model; wherein the multi-task detection model includes a self-supervised contrastive learning model and a few-shot classification model, and the self-supervised contrastive learning model and the few-shot classification model share a feature extractor; the self-supervised contrastive learning model uses a self-supervised contrastive learning unlabeled data set D2 to complete feature extractor training, and the few-shot classification model uses the trained feature extractor to extract features from the few-shot classification labeled data set D1, and uses the extracted image feature map to complete the training of the embedding module in the few-shot classification model and fine-tune the feature extractor; The trained multi-task detection model is used to detect images of power transmission equipment; The self-supervised contrastive learning model uses the self-supervised contrastive learning unlabeled data set D2 to complete the feature extractor training, specifically including: The image samples belonging to the self-supervised contrastive learning unlabeled dataset D2 in the same training batch are input into the pre-trained MASK R-CNN network; the MASK R-CNN network uses the backbone network to extract the image feature map of each image sample; Performing an activation operation on the image feature map of the image sample to obtain a foreground activation map and a background activation map of the image sample; Based on the foreground activation map, background activation map and image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation ; Any two foreground feature representations or two background feature representations constitute a positive pair, and one foreground feature representation and one background feature representation constitute a negative pair; the positive pairs are sorted by feature similarity, and weights are assigned to the positive pairs based on the sorting order value to reduce the interference of low-similarity positive pairs; Design a contrastive learning loss function to pull the representations of positive pairs closer together and pull the representations of negative pairs further apart; Calculate the contrastive learning loss and back-propagate to update the parameters of the MASK R-CNN network.
2. The device defect detection method under a few-sample condition according to claim 1, characterized in that: The few-sample classification annotation dataset D1 includes a support set and a query set, and the image samples in the support set and the query set share the same category.
3. The device defect detection method under a few sample conditions according to claim 1, characterized in that: The step of performing an activation operation on the image feature map of the image sample to obtain a foreground activation map and a background activation map of the image sample specifically includes: The Mask R-CNN network is used to perform semantic segmentation on the image feature map to obtain the segmentation result to distinguish the foreground area and background area of the image sample; Map the foreground area and background area of the segmentation result back to the image feature map to distinguish the feature points belonging to the foreground area and the background area in the image feature map; The feature points of the foreground area are retained, the feature points of the background area are removed, and the channel dimension reduction and normalization operations are performed on the image feature map of the retained foreground area to obtain the foreground activation map of the image sample; at the same time, the feature points of the background area are retained, the feature points of the foreground area are removed, and the channel dimension reduction and normalization operations are performed on the image feature map of the retained background area to obtain the background activation map of the image sample.
4. The device defect detection method under a few sample conditions according to claim 3 is characterized in that: Based on the foreground activation map, the background activation map and the image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation Specifically include: Flatten the foreground activation map and the background activation map into a one-dimensional vector; Flatten the image feature map extracted by the backbone network of the MASK R-CNN network into a single-channel matrix; Based on the tiled foreground activation map, background activation map and image feature map, the image feature map of the image sample is decomposed into a foreground feature representation and background feature representation Among them, Foreground feature representation of samples and background feature representation It can be expressed as: ; In the formula, , , They represent the foreground activation map, background activation map, and image feature map after tiling respectively; and , , .
5. The device defect detection method under a few sample conditions according to claim 4, characterized in that: The method of sorting the positive pairs by using feature similarity and assigning weights to the positive pairs based on the sorting order values to reduce the interference of the positive pairs with low similarity specifically includes: Calculate the cosine similarity of each pair, and then sort the cosine similarities of each foreground pair and background pair respectively, assign large weights to pairs with high similarity, and assign small weights to pairs with low similarity. The weight assignment formula for pairs is as follows: ; represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where A hyperparameter for controlling the smoothness of the exponential function, and Indicates the corresponding sort order value.
6. The device defect detection method under a few-sample condition according to claim 5, characterized in that: The contrast loss function is designed as: ; represents the loss value of the prospect, and Represents the loss value of the background. Represents the overall positive loss value, represents the loss value of the negative pair, represents the overall loss value; where n represents the number of image samples belonging to the unlabeled dataset D2 of the same training batch; represents the weight of two foreground features constituting a positive pair, represents the weight of two background feature representations constituting a positive pair, where yes and The cosine similarity between ; yes and The cosine similarity between yes and The cosine similarity between .
7. The device defect detection method under a few sample conditions according to claim 1, characterized in that: The few-sample classification model uses the trained feature extractor to extract features from the few-sample classification annotated dataset D1, and uses the extracted image feature map to complete the training of the embedding module in the few-sample classification model and the fine-tuning of the feature extractor. Specifically, the following steps are included: Input the image samples belonging to the few-sample classification annotation dataset D1 in the same training batch into the trained MASK R-CNN network; the MASK R-CNN network uses the backbone network to extract the image feature map of the corresponding image sample; The image feature maps corresponding to the support set and the query set are input into the embedding module, which is used to convert the high-dimensional feature matrix into a low-dimensional feature vector; Calculate the average value of feature vectors of multiple image samples of the same category in the support set, and calculate the similarity between the feature vectors of the query set image samples and the average value of feature vectors corresponding to each category in the support set through the distance measurement module; The calculated similarity is assigned to the softmax function to obtain the probability distribution of the query set image samples; Based on the loss between the predicted and true values of the query set image samples, backpropagation updates the embedding module parameters and fine-tunes the feature extractor parameters.
8. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to implement the steps of a device defect detection method under a few-sample condition as claimed in any one of claims 1 to 7 when executing a computer program stored in the memory.
9. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a device defect detection method under a few-sample condition as claimed in any one of claims 1 to 7 are implemented.
