Power equipment defect data labeling method and system based on cross-modal active learning

Abstract

The invention provides an electrical equipment defect data labeling method and system based on cross-modal active learning, and belongs to the field of electrical equipment defect data labeling. S1, electrical equipment defect data is converted into multi-modal data, the multi-modal data is mapped to a unified feature space, and a multi-modal classification model is constructed; s2, part of samples are selected for labeling, and an initial training set is constructed; s3, training the multi-modal classification model, inputting an unlabeled sample in the multi-modal data into the multi-modal classification model, and outputting a prediction label; s4, according to an active learning strategy, selecting samples with annotation values from the multi-modal data for annotation to obtain annotation samples, adding the annotation samples into the initial training set, and repeating the steps S3 to S4 to meet annotation requirements or stop conditions; classifying the defect data of the power equipment by using the trained multi-modal classification model to obtain a final labeling result; the data labeling cost and workload are reduced, and the labeling accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment defect data labeling, and in particular to a power equipment defect data labeling method and system based on cross-modal active learning. Background Art

[0002] Power equipment plays a vital role in power transmission, distribution, control and protection. However, power equipment may malfunction or have hidden dangers during use, which may cause power supply system failures. Insulation aging, external force damage and irresistible natural disasters caused by long-term operation of electrical equipment may cause equipment defects, which will not only cause safety hazards to the equipment itself and users, but also affect the service life of the equipment and the safe operation of the circuit. Therefore, when power equipment fails, it should be handled immediately, otherwise it will cause equipment damage and major accidents.

[0003] The types of power equipment defect data are numerous and complex. Previous supervised learning methods rely on a large number of manual annotations, which is time-consuming and labor-intensive, with low annotation efficiency and high overall annotation costs. In addition, due to insufficient expert experience, the annotation results are easily inconsistent. In addition, most power equipment defect recognition models are trained to predict a fixed set of object categories and cannot predict new object categories in the test set.

[0004] Active learning is a novel semi-supervised learning method that can improve the learning efficiency and generalization ability of the model while reducing the cost and workload of annotating data. For example, the Chinese invention patent application "A Pedestrian Re-identification Method Based on Deep Active Learning and Model Compression" with publication number CN115690833A applies an active learning algorithm to pedestrian re-identification, and selects samples based on the gradient of the network model, so that the selected samples have both uncertainty and diversity. However, pedestrian images usually have high texture consistency and significant features. Pedestrian recognition models are usually trained to capture the shape, posture and other relevant features of the human body, while transformer defect images may contain different types of defects, textures and background noise. Transformer defect classification needs to focus on subtle defects and anomalies on the transformer surface. The prior art does not provide how to apply active learning strategies to the labeling of power equipment defect data, and how to improve the accuracy of power equipment defect data labeling. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to improve the accuracy of power equipment defect data annotation while reducing the cost and workload of power equipment defect annotation data.

[0006] The present invention solves the above technical problems through the following technical solutions: a method for labeling defect data of power equipment based on cross-modal active learning, comprising the following steps:

[0007] Step 1: Convert the power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model;

[0008] Step 2: Select some samples from the multimodal data for annotation and construct an initial training set;

[0009] Step 3: Train the multimodal classification model, input the unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted labels;

[0010] Step 4: Select samples with annotation value from the multimodal data according to the active learning strategy for annotation, obtain annotated samples, add the annotated samples to the initial training set, repeat steps 3 to 4 until the annotation requirements or stop conditions are met, and obtain a trained multimodal classification model;

[0011] Step 5: Use the trained multimodal classification model to classify the power equipment defect data and obtain the final labeling results.

[0012] The present invention combines a multimodal classification model with an active learning strategy. The active learning strategy reduces the number of samples that need to be labeled, making up for the high cost of labeling power equipment defect data. The multimodal classification model can provide better initial classification performance for the active learning strategy, so that the samples selected by active learning can more effectively improve the model performance, thereby improving the accuracy of labeling power equipment defect data when the labeled samples are limited.

[0013] Preferably, the step 1 comprises:

[0014] 1.1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs;

[0015] 1.2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features.

[0016] 1.3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs;

[0017] 1.4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

[0018] Preferably, the calculation formula of the contrast loss function is as follows:

[0019]

[0020] In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

[0021] Preferably, in step 3, the process of inputting unlabeled samples in the multimodal data into the multimodal classification model and outputting predicted labels includes:

[0022] According to the classification labels of the power equipment defect data, the description text of each category is constructed: a photo of a (object), and they are input into the text encoder to obtain text features. The image to be predicted is input into the image encoder to obtain image features, and the scaled cosine similarity between the image features and N text features is calculated. Then, the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

[0023] Preferably, the step 4 comprises:

[0024] 4.1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, and we get:

[0025]

[0026] In formula (2), Z(x, V) represents the nonlinear function that maps the input x to the output of the second-to-last layer of the network;

[0027] 4.2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is:

[0028]

[0029] 4.3. For all true labels y∈{1,…,K}, let have:

[0030]

[0031] Formula (4) shows that the gradient embedding g xThe norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e.

[0032] 4.4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels;

[0033] 4.5. Random subset S t Add to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

[0034] The present invention also provides a power equipment defect data annotation system based on cross-modal active learning, comprising:

[0035] A model building module is used to convert power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model;

[0036] The annotation module is used to select some samples from the multimodal data for annotation and construct the initial training set;

[0037] The training module is used to train the multimodal classification model, input unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted labels;

[0038] The active learning training module is used to select samples with annotation value in the multimodal data according to the active learning strategy for annotation, obtain the labeled samples, add the labeled samples to the initial training set, repeat the training until the annotation requirements or the stopping conditions are met, and obtain the trained multimodal classification model;

[0039] The classification and labeling module is used to classify the power equipment defect data using the trained multimodal classification model to obtain the final labeling results.

[0040] Preferably, the model building module is also used for:

[0041] 1.1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs;

[0042] 1.2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features.

[0043] 1.3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs;

[0044] 1.4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

[0045] Preferably, the calculation formula of the contrast loss function is as follows:

[0046]

[0047] In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

[0048] Preferably, in the training module, the process of inputting unlabeled samples in the multimodal data into the multimodal classification model and outputting predicted labels includes:

[0049] According to the classification labels of the power equipment defect data, the description text of each category is constructed: a photo of a (object), and they are input into the text encoder to obtain text features. The image to be predicted is input into the image encoder to obtain image features, and the scaled cosine similarity between the image features and N text features is calculated. Then, the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

[0050] Preferably, the active learning training module is also used for:

[0051] 4.1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, and we get:

[0052]

[0053] In formula (2), Z(x, V) represents the nonlinear function that maps the input x to the output of the second-to-last layer of the network;

[0054] 4.2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is:

[0055]

[0056] 4.3. For all true labels y∈{1,…,K}, let have:

[0057]

[0058] Formula (4) shows that the gradient embedding g x The norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e.

[0059] 4.4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels;

[0060] 4.5. Random subset S t Add to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

[0061] The advantages provided by the present invention are:

[0062] 1. The present invention combines a multimodal classification model with an active learning strategy. The active learning strategy reduces the number of samples that need to be labeled, making up for the high cost of labeling defect data of power equipment. The multimodal classification model can provide better initial classification performance for the active learning strategy, so that the samples selected by active learning can more effectively improve the model performance, thereby improving the accuracy of labeling defect data of power equipment when the labeled samples are limited.

[0063] 2. The present invention uses multimodal data to represent the defect characteristics of power equipment, and can improve the accuracy of defect identification by considering multiple data sources at the same time. After the multimodal data is input into the multimodal classification module based on the CLIP model, the image features and text features are mapped into the same space, and the semantic connection between the image and text pairs is strengthened through contrast learning, and the alignment and representation learning of cross-modal data are realized, which can make full use of the information of multimodal data and enhance the accuracy of identifying defects in power equipment; a multi-model classification model is constructed based on the CLIP model. The CLIP model can identify new object categories. When encountering defect categories that do not appear in the training set, the multi-modal classification model will have better performance, which improves the multi-modal classification model's ability to recognize new categories of power equipment defects.

[0064] 3. The present invention mainly adopts a deep batch active learning algorithm to select samples with annotation value for annotation. By selecting samples with the largest amount of information, the sample labeling work is reduced and the annotation cost is reduced. The algorithm integrates uncertainty and diversity, and the selected samples have a larger amount of information. The algorithm uses the K-MEANS seeding algorithm, which gives priority to samples that can generate diversity and bring greater changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A principle block diagram of a method for labeling defect data of electric power equipment based on cross-modal active learning provided by an embodiment of the present invention;

[0066] Figure 2 A flow chart of a method for labeling defect data of electric power equipment based on cross-modal active learning provided by an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of a CLIP-based multimodal classification model in a method for labeling defect data of electric power equipment based on cross-modal active learning provided by an embodiment of the present invention;

[0068] Figure 4 A sub-flow chart of a method for labeling defect data of electric power equipment based on cross-modal active learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] Example 1

[0071] like Figure 1 and Figure 2 As shown, this embodiment provides a method for labeling power equipment defect data based on cross-modal active learning, comprising the following steps:

[0072] Step 1: Convert the power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model.

[0073] The specific steps include:

[0074] 1.1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs;

[0075] 1.2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features.

[0076] 1.3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs;

[0077] 1.4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

[0078] The calculation formula of the contrast loss function is as follows:

[0079]

[0080] In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

[0081] The power equipment defect data is converted into multimodal data, and the multimodal data includes N image-text pairs. The present invention constructs a multimodal classification model based on the CLIP model, and inputs the multimodal data into the CLIP model. The CLIP model includes an image encoder (ResNet50) and a text encoder (Text Transformer) to extract N image features and N text features. There are N positive samples, that is, images and texts that truly belong to a pair, and the remaining N 2 -N image-text pairs are negative samples. N image features and N text features are combined in pairs. By calculating the cosine similarity between image features and text features, the CLIP model predicts N 2 The similarity of possible image-text pairs, such as Figure 3 The training goal of the CLIP model is to maximize the similarity of N positive samples and minimize N2 -The similarity of N negative samples, that is, maximize the value of the orange diagonal and minimize the value of other non-diagonal lines. The calculation of cosine similarity is as follows:

[0082] S(I,T)=cos(I,T)=I·T / ||I||·||T||

[0083] Among them, I represents image features and T represents text features. It can be seen that the cosine similarity S of the positive sample i,i The calculation formula is:

[0084] S i，i (I i , T i )=cos(I i , T i )=I i ·T i / ||I i ||·||T i ||

[0085] Cosine similarity S of negative samples i,k The calculation formula is:

[0086] S i，k (I i , T k )=cos(I i , T k )=I i ·T k / ||I i ||·||T k ||

[0087] Among them, i∈[1,N], k∈[1,N].

[0088] In formula (1), the default value of hyperparameter τ is 0.05, and the optimization direction is that the smaller the negative sample similarity of the denominator is, the better. At this time, the value of the fraction is close to 1, and the value of the loss is close to 0. The purpose of formula (1) is to minimize the distance between the positive text and the image, and maximize the distance between the negative text and the image.

[0089] Step 2: Select some samples from the multimodal data for annotation and construct the initial training set.

[0090] Step 3: Train the multimodal classification model, input the unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted label.

[0091] Specifically, a descriptive text for each category is constructed according to the classification labels of the power equipment defect data: a photo of a (object), and they are input into the text encoder to obtain text features, the image to be predicted is input into the image encoder to obtain image features, the cosine similarity of the image features and the scaled N text features is calculated, and then the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

[0092] The CLIP-based multimodal classification model can identify new object categories. Specifically, image features and text features are mapped to the same space, making similar image and text features closer in space. In the inference stage, when the model encounters a new object category, the model compares the image and text features in the same vector space, and calculates the similarity between the image vector and each text vector to find the text description that best matches the image, which is the possible category of the image. Even if the model has not seen a specific object category during training, it can classify it based on semantic relevance. In addition, the defect category of power equipment may be a compound defect, and the zero-shot capability of CLIP can be used to perform multi-label defect classification.

[0093] Considering that CLIP is trained on 400 million sets of image-text pairs, these data mainly cover natural scenes, and the image encoder structure uses a relatively bulky VIT model. Therefore, when facing vertical power defect scenarios, it may be underfitting, that is, when it encounters objects of different categories from those in the training set, the performance may decline. Therefore, the model needs to be fine-tuned for power defect data, but the fine-tuning effect of VIT is affected by the size of the labeled data, and a certain scale of labeled defect data is required to ensure the effect of fine-tuning. If the defect data is small, and considering the timeliness of power equipment defect monitoring, the present invention uses the trained ResNet50 as the backbone network to extract image features. The CNN network is much lower than the VIT model in terms of the magnitude of the training data set, and CNN features are used to jointly represent text features to obtain better generalization ability and higher reasoning efficiency.

[0094] Step 4: Select samples with annotation value from the multimodal data according to the active learning strategy to obtain annotated samples, add the annotated samples to the initial training set, repeat steps 3 to 4 until the annotation requirements or stop conditions are met, and obtain a trained multimodal classification model; Figure 4 As shown in Figure 2, in t cycles, two main calculations are performed: gradient embedding calculation and sampling calculation. In each cycle, for each example x in the unlabeled sample U, the predicted label of example x is calculated. and the loss in The gradient g of the last layer parameterx , and secondly, in the gradient vector {g x :x∈U\S} to select a set of points through K-MEANS++ seeding algorithm and query their labels. The specific steps include:

[0095] 4.1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out is the parameter of the output layer, θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, t is the number of iterations, t≤T, and we get:

[0096]

[0097] In formula (2), f(x; θ) represents a neural network, and Z(x, V) represents a nonlinear function that maps the input x to the output of the second-to-last layer of the network;

[0098] 4.2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is:

[0099]

[0100] 4.3. For all true labels y∈{1,…,K}, let have:

[0101]

[0102] Formula (4) shows that the gradient embedding g x The norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e. This shows that the gradient embedding g x The norm of conservatively estimates the impact of the sample on the current model. If the current model θ is very confident about the example x, that is, the vector p is biased towards the standard basis vector e j ,but And the vector The length of is small. Therefore, the gradient embedding g x The length of will also be small. Such high-confidence samples tend to have smaller gradient embeddings and are unlikely to be repeatedly selected in the tth iteration of K-MEANS++.

[0103] 4.4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels;

[0104] 4.5. Random subset S t Add to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

[0105] The success of deep learning is inseparable from a large amount of labeled data, but the cost of labeling a large amount of power defect data is high. In order to reduce the labeling cost and improve the effect of the model, the present invention uses deep active learning to train the multimodal classification model. The multimodal classification model is trained by selecting samples with the largest amount of information and the least labeling work, thereby greatly reducing the number of samples required to be labeled and reducing the labeling cost. Active learning selects a batch of the most useful samples through a certain query function, and asks experts for labels, and then uses the newly labeled sample set obtained to train the classification model and conduct the next round of queries.

[0106] Among various active learning methods, the commonly used strategies for query functions are uncertainty criteria and diversity criteria. The greater the uncertainty, the greater the information entropy and the richer the information contained; and the greater the diversity, the more comprehensive the selected samples can represent the entire data set. The present invention adopts a method that combines uncertainty and diversity, taking into account the advantages of both, that is, selecting a batch of the most useful samples that contain rich information and can fully represent the entire data set.

[0107] The present invention characterizes the uncertainty of the model to the sample by the size of the gradient norm, that is, when the model's prediction of a certain sample causes the gradient of the loss function relative to the model parameters to be large, thus requiring a large update of the model, the model is uncertain about the sample. Diversity is represented by gradients, and gradient vectors are used for clustering. When samples with large differences are selected, the model parameters are updated in different directions. This gradient embedding method can not only convey uncertainty information about the model, but also reflect the possible update direction after receiving the sample label.

[0108] After expressing uncertainty and diversity with gradients, active learning uses a corresponding sampling algorithm to select a batch of samples with high uncertainty and diversity. In order to enable the model to better adapt to large-scale and diverse changes, the present invention adopts the K-MEANS seeding algorithm, which aims to give priority to samples that can generate diversity and bring large changes, rather than selecting a batch of similar samples, which may reduce the uncertainty of the remaining samples after obtaining a batch of labels. The K-MEANS seeding algorithm selects centroids through iterative sampling, and these points are proportional to the squared distance of the nearest centroid that has been selected.

[0109] Step 5: Use the trained multimodal classification model to classify the power equipment defect data and obtain the final labeling results.

[0110] The present invention combines a multimodal classification model based on CLIP with an active learning strategy. The active learning strategy reduces the number of samples that need to be labeled, making up for the high cost of labeling power equipment defect data. Compared with ordinary classification models, the multimodal classification model can provide better initial classification performance for the active learning strategy, so that the samples selected by active learning can more effectively improve the model performance. The power equipment defect data is labeled using the power equipment defect data labeling method based on cross-modal active learning of the present invention. The labeling results have high consistency and reliability, which can provide a reliable basis for power equipment defect detection and analysis.

[0111] By using multimodal data to represent the defect characteristics of power equipment and considering multiple data sources at the same time, the accuracy of defect recognition can be improved. After the multimodal data is input into the multimodal classification module based on CLIP, the image features and text features are mapped into the same space. The semantic connection between the image and text pairs is strengthened through contrastive learning, and the alignment and representation learning of cross-modal data are realized. The information of multimodal data can be fully utilized, and the accuracy of identifying power equipment defects is enhanced. On the other hand, since the CLIP model can recognize new object categories, when encountering defect categories that do not appear in the training set, the multimodal classification model based on the CLIP model will have better performance, improving the multimodal classification model's ability to recognize new categories of power equipment defects.

[0112] The deep batch active learning algorithm is used to select samples with annotation value for annotation. By selecting samples with the largest amount of information, the sample labeling work is reduced, and the annotation cost is reduced to minimize the workload of manual annotation, reduce the annotation cost, and improve the annotation efficiency. Active learning integrates uncertainty and diversity, and the selected samples have a larger amount of information. By using the K-MEANS seeding algorithm, samples that can generate diversity and bring greater changes are preferentially selected.

[0113] Example 2

[0114] This embodiment provides a power equipment defect data annotation system based on cross-modal active learning, including:

[0115] The model building module is used to convert the power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model.

[0116] The model building module is also used to:

[0117] 1.1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs;

[0118] 1.2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features.

[0119] 1.3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs;

[0120] 1.4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

[0121] The calculation formula of the contrast loss function is as follows:

[0122]

[0123] In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

[0124] The annotation module is used to select some samples from the multimodal data for annotation and construct the initial training set;

[0125] The training module is used to train the multimodal classification model, input unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted labels.

[0126] Specifically, a descriptive text for each category is constructed according to the classification labels of the power equipment defect data: a photo of a (object), and they are input into the text encoder to obtain text features, the image to be predicted is input into the image encoder to obtain image features, the cosine similarity of the image features and the scaled N text features is calculated, and then the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

[0127] The active learning training module is used to select samples with annotation value in multimodal data according to the active learning strategy for annotation, obtain annotated samples, add the annotated samples to the initial training set, and repeat the training until the annotation requirements or the stopping conditions are met, thereby obtaining a trained multimodal classification model.

[0128] The active learning training module is also used to:

[0129] 4.1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, and we get:

[0130]

[0131] In formula (2), Z(x, V) represents the nonlinear function that maps the input x to the output of the second-to-last layer of the network;

[0132] 4.2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is:

[0133]

[0134] 4.3. For all true labels y∈{1,…,K}, let have:

[0135]

[0136] Formula (4) shows that the gradient embedding g x The norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e.

[0137] 4.4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels;

[0138] 4.5. Random subset S tAdd to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

[0139] The classification and labeling module is used to classify the power equipment defect data using the trained multimodal classification model to obtain the final labeling results.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for labeling defect data of power equipment based on cross-modal active learning, characterized by: The following steps are involved: Step 1: Convert the power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model; Step 2: Select some samples from the multimodal data for annotation and construct an initial training set; Step 3: Train the multimodal classification model, input the unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted labels; Step 4: Select samples with annotation value from the multimodal data according to the active learning strategy for annotation, obtain annotated samples, add the annotated samples to the initial training set, repeat steps 3 to 4 until the annotation requirements or stop conditions are met, and obtain a trained multimodal classification model; Step 5: Use the trained multimodal classification model to classify the power equipment defect data and obtain the final labeling results.

2. The method for labeling defect data of power equipment based on cross-modal active learning according to claim 1 is characterized in that: The step one comprises: 1.

1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs; 1.

2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features. 1.

3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs; 1.

4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

3. The method for labeling defect data of power equipment based on cross-modal active learning according to claim 2 is characterized in that: The calculation formula of the contrast loss function is as follows: In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

4. The method for labeling defect data of power equipment based on cross-modal active learning according to claim 1 is characterized in that: In step 3, the process of inputting unlabeled samples in the multimodal data into the multimodal classification model and outputting predicted labels includes: According to the classification labels of the power equipment defect data, the description text of each category is constructed: a photo of a (object), and they are input into the text encoder to obtain text features. The image to be predicted is input into the image encoder to obtain image features, and the scaled cosine similarity between the image features and N text features is calculated. Then, the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

5. The method for labeling defect data of power equipment based on cross-modal active learning according to claim 2 is characterized in that: The fourth step comprises: 4.

1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, and we get: In formula (2), Z(x, V) represents the nonlinear function that maps the input x to the output of the second-to-last layer of the network; 4.

2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is: 4.

3. For all true labels y∈{1,…,K}, let have: Formula (4) shows that the gradient embedding g x The norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e. 4.

4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels; 4.

5. Random subset S t Add to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

6. The power equipment defect data annotation system based on cross-modal active learning is characterized by: include: A model building module is used to convert power equipment defect data into multimodal data, map the multimodal data into a unified feature space, and build a multimodal classification model; The annotation module is used to select some samples from the multimodal data for annotation and construct the initial training set; The training module is used to train the multimodal classification model, input unlabeled samples in the multimodal data into the multimodal classification model, and output the predicted labels; The active learning training module is used to select samples with annotation value in the multimodal data according to the active learning strategy for annotation, obtain the labeled samples, add the labeled samples to the initial training set, repeat the training until the annotation requirements or the stopping conditions are met, and obtain the trained multimodal classification model; The classification and labeling module is used to classify the power equipment defect data using the trained multimodal classification model to obtain the final labeling results.

7. The power equipment defect data annotation system based on cross-modal active learning according to claim 6 is characterized in that: The model building module is also used to: 1.

1. Collect power equipment defect data and convert the power equipment defect data into multimodal data of N image-text pairs; 1.

2. Input the multimodal data into the CLIP model and use the image encoder and text encoder to extract N image features and N text features. 1.

3. Combine N image features and N text features into the CLIP model and predict N 2 Similarity of image-text pairs; 1.

4. Train the CLIP model. When the contrast loss function is minimized, the trained CLIP model is obtained. A multimodal classification model is constructed based on the trained CLIP model.

8. The power equipment defect data annotation system based on cross-modal active learning according to claim 7 is characterized in that: The calculation formula of the contrast loss function is as follows: In formula (1), S i,i Represents the cosine similarity of positive samples, S i,k represents the cosine similarity of negative samples, and τ is a hyperparameter.

9. The power equipment defect data annotation system based on cross-modal active learning according to claim 6 is characterized in that: In the training module, the process of inputting unlabeled samples in the multimodal data into the multimodal classification model and outputting predicted labels includes: According to the classification labels of the power equipment defect data, the description text of each category is constructed: a photo of a (object), and they are input into the text encoder to obtain text features. The image to be predicted is input into the image encoder to obtain image features, and the scaled cosine similarity between the image features and N text features is calculated. Then, the category corresponding to the text with the largest similarity is selected as the prediction result of image classification.

10. The power equipment defect data annotation system based on cross-modal active learning according to claim 7, characterized in that: The active learning training module is also used to: 4.

1. Definition of p i =f(x;θ) i , f is parameterized by θ = (W, V), θ out =W = (W1, ..., W K ) T ∈R K×d is the weight of the last layer, V is composed of the weights of all previous layers, x is an example of unlabeled samples in multimodal data, x is used as input, K is the total number of categories, and we get: In formula (2), Z(x, V) represents the nonlinear function that maps the input x to the output of the second-to-last layer of the network; 4.

2. For the true label y, define is the gradient embedding in the algorithm, where the predicted label Then the gradient of label i is: 4.

3. For all true labels y∈{1,…,K}, let have: Formula (4) shows that the gradient embedding g x The norm of is a lower bound on the norm of the loss gradient induced by examples with true label y with respect to the weights of the last layer, i.e. 4.

4. In the gradient vector {g x :x∈U\S}, the random subset S of unlabeled samples U in the multimodal data is calculated by K-MEANS++ seeding algorithm t , and query the random subset S t Labels; 4.

5. Random subset S t Add to the initial training set S, and train the multimodal classification model based on the updated training set. When the expected value of the loss function E s [l CE When (f(x;θ),y)] is the smallest, the trained multimodal classification model is obtained.

Images