Power transmission line defect identification method based on multi-modal comparative learning

Abstract

The invention provides a power transmission line defect identification method based on multi-modal comparative learning, and the method comprises the steps: constructing a power transmission line image-text-label data set, and achieving the recognition of defects of a power transmission line while employing the powerful pre-training representation in an existing vision-language model; the electric power semantic knowledge contained in the text modal data is introduced through a specific auxiliary pre-training task in the electric power field, and the accuracy of power transmission line defect recognition is effectively improved. Meanwhile, a fine tuning strategy based on comparative learning is designed, I TC is added on the basis of downstream transmission line defect identification tasks, the whole identification model is finely tuned by using two tasks to relieve the overfitting problem in the fine tuning stage, and a final transmission line defect identification model is obtained through training.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method for identifying defects in power transmission lines based on multimodal contrast learning. Background Art

[0002] The stable operation of transmission lines is closely related to maintaining the safety of power systems and ensuring the quality of power supply. Due to long-term exposure to harsh outdoor environments and the combined effects of multiple external factors such as wind, rain, snow and thunder, the hardware in the transmission lines is prone to defects and failures of varying degrees, posing a serious threat to the safety of the power system. Due to the long distances of transmission lines and the fact that some towers are located in complex terrain environments, traditional manual inspection methods require a lot of manpower costs and are inefficient, and can no longer meet the needs of new power system construction. With the rapid development of transmission line inspection technology based on drones in recent years, the current transmission line defect identification method mainly uses computer vision technology based on deep learning to process and analyze aerial images of transmission lines. This inspection mode effectively improves the intelligence level of power grid operation and maintenance, and also effectively reduces the burden of transmission line inspection operations.

[0003] However, the existing deep learning-based transmission line defect recognition model mainly uses the traditional supervised pre-training model for transfer learning. Restricted by the accessible categories and the scale of pre-training data, the generalization ability of the traditional supervised pre-training model is relatively limited, and it lacks the power semantic knowledge unique to the power field. The research on the visual-language model, which has received widespread attention in recent years, provides ideas for overcoming the above problems. Due to the pre-training on massive image-text data at the network level, the visual-language model has a stronger generalization ability than the traditional supervised pre-training model, providing a better training starting point for training the deep learning-based transmission line defect recognition model. However, due to the existence of data barriers in the power industry, the existing visual-language model can only access limited power-related data during the pre-training stage and lacks power semantic knowledge. There are still the following two problems when it needs to be applied to the transmission line defect recognition task:

[0004] 1. The existing visual-language model lacks power semantic knowledge, and its corresponding pre-training method does not fully consider the unique inter-class relationship within the transmission line inspection data. Therefore, the effect of fine-tuning the existing visual-language model directly on the transmission line defect recognition data is poor. Even if the existing visual-language model is further pre-trained and fine-tuned on the power field data, it can only achieve suboptimal results.

[0005] 2. Due to the limited data in the power field, the existing vision-language model inevitably has a certain overfitting problem after domain-specific pre-training. When learning the transmission line defect recognition task, using supervised learning objectives to fine-tune the model will further aggravate the overfitting problem, thereby limiting the recognition accuracy of the model in the transmission line defect recognition task.

[0006] Therefore, under the above background, the high-performance two-stream vision-language model CN-CLIP is applied to the task of transmission line defect recognition. According to the basic characteristics of transmission line inspection data, the problems existing in the original vision-language model in the transmission line application scenario are improved, and a transmission line defect recognition model with higher accuracy is trained, which has important research significance for the intelligent inspection of transmission lines. Summary of the invention

[0007] The purpose of the present invention is to design a method for identifying power line defects based on multimodal contrastive learning, so as to overcome the adverse effects on recognition accuracy caused by the lack of power semantic knowledge in the existing visual-language model, the visual-language model pre-training method ignoring the internal inter-class relationship of the transmission line inspection data, and the aggravation of overfitting in the process of fine-tuning the visual-language model. The present invention designs two field-specific auxiliary pre-training tasks to help the model better distinguish the semantics of different categories and understand the abstract concept of defect-normal, and effectively alleviates the overfitting problem in the fine-tuning stage through a fine-tuning strategy based on contrastive learning, thereby improving the accuracy of power line defect identification.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for identifying defects in a power transmission line based on multimodal contrastive learning, the method comprising the following steps:

[0010] S1, constructs a transmission line image-text-label dataset based on transmission line inspection images, which contains image-text-label triplets related to various transmission line components and defects, among which the power semantic knowledge contained in the text data;

[0011] S2, select the high-performance two-stream vision-language model CN-CLIP as the baseline model, and use the existing general pre-trained representation in it to reduce training costs and improve training efficiency;

[0012] S3, design a pre-training algorithm unique to the power field, using I TC as the basic pre-training task, while avoiding destroying the original pre-training representation, and introducing power semantic knowledge into the model;

[0013] S4, based on I TC, CTM is designed as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0014] S5, based on I TC, DNC is designed as an auxiliary pre-training task to help the model understand the abstract concept of defect-normal more fully, so as to identify defects more effectively;

[0015] S6, using ITC, CTM and DNC to pre-train a visual basic model for the task of transmission line defect recognition, and using the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage;

[0016] S7, designed a fine-tuning strategy based on contrastive learning, added ITC on the basis of the downstream transmission line defect recognition task, used the two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and trained the final transmission line defect recognition model;

[0017] The design of a pre-training algorithm unique to the power field uses I TC as the basic pre-training task, while avoiding destroying the original pre-training representation, and introduces power semantic knowledge into the model, specifically including:

[0018] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0019] V=f i (I); L = f t (T) (1)

[0020] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, respectively. V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively.

[0021] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0022]

[0023] Where n is the batch size, τ is the temperature coefficient, and v i and v j represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are the contrastive learning losses from image to text and from text to image, respectively, and l ITC is the total loss of ITC.

[0024] An electronic device comprises: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the above-mentioned transmission line defect identification method based on multimodal contrast learning.

[0025] A storage medium, characterized in that the storage medium stores computer executable instructions, and the computer executable instructions are used to execute the above-mentioned transmission line defect identification method based on multimodal contrast learning.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The embodiment of the present invention discloses a method for identifying power line defects based on multimodal contrastive learning, comprising the steps of: constructing a power line image-text-label data set, comprising a plurality of power line components and defect-related image-text-label triplets, for pre-training a visual basic model for a power line defect recognition task; selecting a dual-stream visual-language model CN-CLIP as a baseline model; designing a pre-training algorithm specific to the power field, taking image-text contrast learning (ITC) as a basic pre-training task, and introducing the power semantic knowledge contained in the power line image-text-label data set into the model without destroying the original pre-training representation; designing two auxiliary pre-training tasks, component type matching (CTM) and defect-normality comparison (DNC), based on ITC, making full use of the unique inter-class relationship within the power line inspection data to help the model distinguish the semantics of different categories more effectively and understand the abstract concept of defect-normal more fully; using ITC to TC, CTM and DNC pre-train a visual basic model for the task of transmission line defect recognition, and use the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage; design a fine-tuning strategy based on contrastive learning, add I TC on the basis of the downstream transmission line defect recognition task, use two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and train the final transmission line defect recognition model. The present invention applies the CN-CLIP algorithm to transmission line defect recognition, and can effectively improve the accuracy of transmission line defect recognition by introducing the power semantic knowledge contained in text modal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly demonstrate the embodiments of the present invention or the technical solutions in the prior art, the drawings involved in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 It is a flow chart of a method for identifying power transmission line defects based on multimodal contrast learning according to an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of the execution of the CTM pre-training task according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the execution of the DNC pre-training task according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of the overall training process of an embodiment of the present invention;

[0033] Figure 5 A thermal map of the classification of transmission line defects according to an embodiment of the present invention;

[0034] Figure 6 This is a diagram showing the effect of power transmission line defect detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The purpose of the present invention is to provide a method for identifying defects in power lines based on multimodal contrastive learning, so as to solve the problems that the existing visual-language model lacks power semantic knowledge, the visual-language model pre-training method ignores the internal inter-class relationship of transmission line inspection data, and the overfitting in the process of fine-tuning the visual-language model is aggravated, so as to achieve the purpose of improving the accuracy of transmission line defect identification.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1As shown, a flow chart of a method for identifying power transmission line defects based on multimodal contrast learning provided by the present invention comprises the following steps:

[0039] S1, constructs a transmission line image-text-label dataset based on transmission line inspection images, which contains image-text-label triplets related to various transmission line components and defects, among which the power semantic knowledge contained in the text data;

[0040] S2, select the high-performance two-stream vision-language model CN-CL IP as the baseline model, and use the existing general pre-trained representation in it to reduce training costs and improve training efficiency;

[0041] S3, design a pre-training algorithm unique to the power field, using I TC as the basic pre-training task, while avoiding destroying the original pre-training representation, and introducing power semantic knowledge into the model;

[0042] S4, based on I TC, CTM is designed as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0043] S5, based on I TC, DNC is designed as an auxiliary pre-training task to help the model understand the abstract concept of defect-normal more fully, so as to identify defects more effectively;

[0044] S6, using ITC, CTM and DNC to pre-train a visual basic model for the task of transmission line defect recognition, and using the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage;

[0045] S7, designs a fine-tuning strategy based on contrastive learning, adds ITC on the basis of the downstream transmission line defect recognition task, uses the two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and trains to obtain the final transmission line defect recognition model.

[0046] Since traditional supervised pre-training methods generally use image-label data containing limited concepts and image classification as pre-training tasks to pre-train the model, the resulting model has limited generalization ability and lacks power semantic knowledge. Therefore, when it is transferred to transmission line inspection data, it can only achieve suboptimal results; while multimodal pre-training uses a large amount of image-text data, generally using I TC as a pre-training task to pre-train the model, so that the model has strong generalization ability and can also explicitly introduce the power semantic knowledge contained in the text modal data into the model, so it has more advantages in the power field than traditional supervised pre-training methods. Multimodal pre-training requires a large number of instance-level images and instance-level text samples as support, so it is necessary to crop each component and defect instance from the original inspection image containing multiple components and defect instances into a separate instance-level image, and generate matching instance-level text. In addition, due to the needs of pre-training algorithms unique to the power field, it is also necessary to retain the category label corresponding to each instance-level image, so this method uses image-text-label data to pre-train the model. Therefore, in the step S1, a transmission line image-text-label dataset is constructed based on the transmission line inspection image, which includes image-text-label triplets related to various transmission line components and defects, specifically including:

[0047] Firstly, the original transmission line inspection images are annotated and cropped to obtain instance level images of various transmission line components and defects, each of which is accompanied by a corresponding category label.

[0048] Then, the category labels corresponding to the instance level images are used to fill in the predefined templates or match the predefined descriptions, and a piece of instance level text is generated for each instance level image, completing the construction of the transmission line image-text-label dataset.

[0049] In this method, considering that the original CN-CLIP model does not contain power semantic knowledge, and training the visual-language model from scratch requires billions of data and a large amount of computing power, while the available power transmission line inspection data and computing power resources are difficult to meet this requirement. Therefore, in step S2, the high-performance dual-stream visual-language model CN-CLIP is selected as the baseline model, and the existing general pre-trained representation is used to reduce the training cost and improve the training efficiency, specifically including:

[0050] Build the CN-CLIP model, and based on the idea of ​​transfer learning, use open source pre-trained weights to initialize the model parameters to avoid the problem of training the model from scratch requiring a large amount of training data and computing resources.

[0051] In this method, considering that the original CN-CLIP model is obtained by pre-training the ITC task on a large amount of image-text data at the network level, only using other pre-training tasks in the pre-training specific to the power field will destroy the original pre-training representation. Therefore, in the step S3, a pre-training algorithm specific to the power field is designed, using ITC as the basic pre-training task, while avoiding destroying the original pre-training representation, the power semantic knowledge is introduced into the model, specifically including:

[0052] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0053] V=f i (I); L = f t (T) (1)

[0054] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, respectively. V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively.

[0055] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0056]

[0057] Where n is the batch size, τ is the temperature coefficient, and v i and v j represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are the contrastive learning losses from image to text and from text to image, respectively, and l ITC is the total loss of ITC.

[0058] In this method, considering that there are two attributes, component type and component status, implicitly in the transmission line inspection data, the transmission line inspection data has a more complex category relationship than the general domain data. Specifically, the category relationship between two instances in the general domain data can be summarized as "same" and "different", while there are three categories of category relationships between two instances in the transmission line inspection data, namely: "same component type and same status", "same component type but different status" and "different types of components". For example: the category relationship between two normal equalizing ring instances is "same component type and same status"; the category relationship between a normal equalizing ring instance and a defective equalizing ring instance is "same component type but different status"; and the category relationship between any equalizing ring instance and any shielding ring instance is "different types of components". Distinguishing the above three category relationships by training the model helps to better distinguish categories with similar semantics. Therefore, in the step S4, CTM is designed as an auxiliary pre-training task based on I TC to help the model more effectively distinguish the semantics of different component types and component status to avoid confusion with similar categories, specifically including:

[0059] CTM is performed using the image feature matrix and text feature matrix obtained in ITC, which includes four subtasks: Image-Text Type Matching (ITTM), Text-Image Type Matching (TITM), Image-Image Type Matching (IITM), and Text-Text Type Matching (TTTM). Taking ITTM as an example, first, shuffle the text feature matrix and label vector:

[0060] L′,y′=shuffle(L,y) (5)

[0061] Where shuffle(·) is the shuffle function, y is the label vector composed of category labels, L' and y' are the shuffled text feature matrix and shuffled label vector, respectively, and both share the same random number seed.

[0062] Secondly, a mapping matrix shared by the four subtasks is used to map the joint feature matrix obtained by concatenating the shuffled text feature matrix and the image feature matrix. After normalization by row, the final matching result matrix is ​​obtained:

[0063] P ITTM =Softmax(Concat(V,L′)·W CTM ) (6)

[0064] Where Concat(·,·) is the concatenation function, W CTM is the mapping matrix, P ITTM is the matching result matrix.

[0065] Again, by comparing the label vector and the shuffled label vector element by element, we determine whether the category relationship between the two elements at the corresponding positions belongs to the same component type and the same state, the same component type but different states, and different types of components, and obtain the label matrix of ITTM:

[0066] y i ITTM =φ(y i ,y i ′) (7)

[0067] In the formula, y i ITTM is the relationship label corresponding to the i-th group of samples in the joint feature matrix, y i and i ′ are the category labels of the i-th sample in the label vector and the shuffled label vector, respectively, and φ(·,·) is the relationship judgment function.

[0068] Then, the loss of ITTM is calculated as follows:

[0069]

[0070] In the formula, is the jth element of the relationship label corresponding to the i-th group of samples in the joint feature matrix, The jth element in the i-th row of the matching result matrix, l ITTM For ITTM's loss.

[0071] Finally, the average of the four CTM subtask losses is taken as the total loss of CTM:

[0072] l CTM =(l ITTM +l TITM +l IITM +l TTTM ) / 4 (9)

[0073] In the formula, l TITM , l IITM and l TTTM are the losses of TITM, IITM and TTTM tasks respectively, l CTM is the total loss of CTM. The execution process of the four CTM subtasks is as follows Figure 2 As shown, by learning four CTM subtasks, the model can align the semantic information about various parts and their states in the visual and language modalities, thereby better distinguishing different categories.

[0074] In this method, it is considered that understanding the abstract concept of defect-normal is of vital importance to the downstream defect identification task, and the traditional pre-training tasks have ignored the implicit component status attributes in the transmission line inspection data, and therefore cannot help the model learn the abstract concept of defect-normal. This method uses the characteristics that the defect and normal categories under each type of component in the transmission line inspection data exist in pairs or groups (such as: the equalizing ring component type contains two categories of "normal equalizing ring" and "broken equalizing ring"). By comparing the defect and normal category samples under the same component type, the model can better recognize the defect-normal concept in this type of component. Therefore, in the step S5, DNC is designed as an auxiliary pre-training task based on I TC to help the model more fully understand the abstract concept of defect-normal, so as to more effectively identify defects, specifically including:

[0075] The DNC is performed using the image feature matrix obtained in ITC. The number of subtasks included is determined by the number of component types with both defective and normal categories in the power line image-text-label dataset. Taking the grading ring contrast (GRD) subtask as an example, its execution process is as follows: Figure 3 First, according to the label vector and the category index of the defective and normal equalizing rings, the feature vectors corresponding to the defective equalizing rings and normal equalizing rings are screened out from the image feature matrix:

[0076]

[0077] Where σ(·,·,·) is the screening function, i DGR and i NGR are the category indexes of defective and normal grading rings, V DGR and V NGR are the defective balancing ring feature matrix and the normal balancing ring feature matrix composed of the feature vectors corresponding to the defective balancing ring and the normal balancing ring samples, respectively. and They represent the feature vector of the i-th defective balancing ring and the feature vector of the j-th normal balancing ring respectively, and k and q are the numbers of defective balancing rings and normal balancing rings in the training batch respectively.

[0078] Secondly, the defective pressure equalizing ring feature matrix and the normal pressure equalizing ring feature matrix are concatenated to obtain the defective-normal pressure equalizing ring feature matrix containing all the pressure equalizing ring features, and the cosine similarity between the defective-normal pressure equalizing ring feature matrix and itself is calculated. After normalization, the similarity score matrix can be obtained:

[0079] V GR =Concat(V DGR ,VNGR ) (12)

[0080]

[0081] Where V GR represents the defect-normal grading ring characteristic matrix, S GR is the similarity score matrix, represents the similarity score between the i-th pressure equalizing ring and the j-th pressure equalizing ring, and m is the total number of pressure equalizing rings in the training batch.

[0082] Again, the label matrix of GRC is generated according to the number of defective equalizing rings and normal equalizing rings in the training batch. The formula is:

[0083]

[0084] In the formula, Z GR is the label matrix of GRC, is the similarity label between the i-th pressure equalizing ring and the j-th pressure equalizing ring.

[0085] Then, the loss of GRC is calculated as follows:

[0086]

[0087] In the formula, l GRC For the loss of GRC.

[0088] Finally, the average of all DNC subtask losses is taken as the total loss of DNC:

[0089]

[0090] In the formula, l DNC(i) represents the loss of the ith subtask of DNC, t is the number of subtasks of DNC, l DNC is the total loss of DNC.

[0091] The present invention integrates the three pre-training tasks mentioned in steps S3, S4 and S5 into the power field-specific pre-training of the baseline model CN-CLIP mentioned in step S2, forming a complete set of power field-specific pre-training algorithms, such as Figure 4 Shown on the left.

[0092] In step S6, a visual basic model for the task of power transmission line defect recognition is pre-trained using IT CTM and DNC, and the image encoder therein is used as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage, specifically including:

[0093] First, the three pre-training tasks mentioned above are used to perform pre-training on CN-CLIP specifically for the power field, and a visual basic model for the task of transmission line defect recognition is obtained. The total loss calculation formula in the pre-training stage is:

[0094] l pre-train =λ1l ITC +λ2l CTM +λ3l DNC (18)

[0095] In the formula, l pre-train is the total loss in the pre-training stage, and λ1, λ2, and λ3 are used to adjust the coefficients of the proportion of the three losses in the total loss.

[0096] Then, the image encoder in the visual base model for the power transmission line defect recognition task is used as the backbone network of the downstream defect recognition model, connected to the task-specific head network for performing the defect recognition task, and the downstream defect recognition model is constructed.

[0097] In this method, considering the limited scale of transmission line inspection data, the visual basic model obtained by pre-training for the transmission line defect recognition task inevitably has the problem of overfitting. According to the traditional fine-tuning paradigm, the fine-tuning process of the model guided by the supervised learning target with a strong supervisory signal will further aggravate the overfitting, while the weak supervisory signal provided by the contrast learning target used in pre-training can alleviate the overfitting to a certain extent. Therefore, in the step S7, a fine-tuning strategy based on contrastive learning is designed, and I TC is added on the basis of the downstream transmission line defect recognition task. The two tasks are used to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and the final transmission line defect recognition model is trained, which specifically includes:

[0098] When the downstream transmission line defect recognition task is a defect classification task, first, the image feature matrix output by the image encoder is passed into the task-specific head network to obtain the defect classification task output. The corresponding classification loss calculation formula is:

[0099]

[0100] In the formula, is the label of the i-th sample belonging to the j-th category, is the model's predicted probability that the i-th sample belongs to the j-th category, c is the category of the defect classification task, and l CLS is the classification loss.

[0101] Secondly, fill the predefined template or match the predefined description with the category label of the downstream defect classification dataset, generate a corresponding text for each image, pass all the text into the text encoder to obtain the text feature matrix, and calculate the ITC loss according to formula (2)-formula (4). The total loss in the fine-tuning stage is the sum of the losses of the two tasks, and the formula is:

[0102] l fine-tune = l CLS +l ITC (20)

[0103] In the formula, l fine-tune is the total loss in the fine-tuning stage.

[0104] When the downstream transmission line defect recognition task is a defect detection task, first, the instances in the downstream defect detection dataset are cropped into instance level images of various scales according to the annotation boxes. Then, the backbone network of the defect detection model and the classification branch of the task-specific head network are fine-tuned according to the method of the downstream transmission line defect recognition task as a defect classification task. The loss at this stage is calculated as:

[0105]

[0106] In the formula, is the fine-tuning loss of the first stage.

[0107] Then, ITC is removed and the entire defect detection model is fine-tuned using two tasks, defect classification and bounding box regression, on the original downstream defect detection dataset. The loss at this stage is calculated as:

[0108]

[0109] In the formula, is the fine-tuning loss of the second stage, l BBOX is the bounding box regression loss.

[0110] Finally, the text encoder is removed to obtain the final transmission line defect recognition model. This fine-tuning strategy can be used Figure 4 The right side summarizes the representation. During model inference, the image encoder and the task-specific head network are used to infer the test image to obtain the transmission line defect recognition output, which is:

[0111] Output = Taskhead(f i (x)) (23)

[0112] Where x is the test image input, Output is the transmission line defect recognition output, and Taskhead(·) is the task-specific head network.

[0113] The thermal map and detection effect generated by the method of the present invention in the task of identifying defects in power transmission lines are as follows: Figure 5 and Figure 6 As shown. The present invention designs a pre-training algorithm specific to the power field based on the dual-stream vision-language model CN-CL IP, and uses ITC as the basic pre-training task to avoid the destruction of the original pre-training representation; CTM is designed as an auxiliary pre-training task based on I TC, and the model is trained to distinguish the category relationship between any two instances in the transmission line inspection data to help the model more effectively distinguish the semantics of different categories; DNC is designed as another auxiliary pre-training task based on I TC, and the model is trained to compare the defect and normal category samples under the same component type to help the model more fully understand the abstract concept of defect-normal; in view of the aggravated overfitting problem of the visual basic model for the transmission line defect recognition task obtained by pre-training in the fine-tuning stage, a fine-tuning strategy based on contrastive learning is designed, and I TC is added on the basis of the downstream transmission line defect recognition task, and a contrastive learning target with a weak supervisory signal is used to alleviate the overfitting problem in the fine-tuning stage. It can be seen that the method described in the present invention can help the transmission line defect recognition model learn the transmission line-related semantic information while understanding the abstract normal-defect visual concept. The present invention effectively introduces the power semantic knowledge contained in the text modal data into the transmission line defect recognition model, thereby effectively improving the recognition accuracy of transmission line defects.

[0114] At the same time, the present application also provides a device corresponding to the method for identifying power transmission line defects based on multimodal contrast learning, as well as an electronic device and a storage medium, which are as follows:

[0115] The device comprises:

[0116] The S1 module is used to construct a transmission line image-text-label dataset based on transmission line inspection images, which contains image-text-label triplets related to various transmission line components and defects, including the power semantic knowledge contained in the text data;

[0117] The S2 module is used to select the high-performance two-stream vision-language model CN-CLIP as the baseline model and use the existing general pre-trained representations in it to reduce training costs and improve training efficiency;

[0118] The S3 module is used to design a pre-training algorithm specific to the power sector, using I TC as the basic pre-training task to introduce power semantic knowledge into the model while avoiding destroying the original pre-training representation;

[0119] The S4 module is used to design CTM based on I TC as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0120] The S5 module is used to design DNC as an auxiliary pre-training task based on I TC to help the model understand the abstract concept of defect-normal more fully, so as to identify defects more effectively;

[0121] The S6 module is used to pre-train a visual basic model for the task of transmission line defect recognition using ITC, CTM and DNC, and use the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage;

[0122] The S7 module is used to design a fine-tuning strategy based on contrastive learning. It adds ITC to the downstream transmission line defect recognition task, uses the two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and trains the final transmission line defect recognition model.

[0123] The S3 module is specifically used for:

[0124] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0125] V=f i (I); L = f t (T) (1)

[0126] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, respectively. V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively.

[0127] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0128]

[0129] Where n is the batch size, τ is the temperature coefficient, and v i and v j represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are the contrastive learning losses from image to text and from text to image, respectively, and l ITC is the total loss of ITC.

[0130] An electronic device comprises: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the transmission line defect identification method based on multimodal contrast learning.

[0131] A storage medium stores computer executable instructions for executing a method for identifying power transmission line defects based on multimodal contrast learning.

[0132] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for identifying power transmission line defects based on multimodal contrastive learning, characterized in that: The method comprises the following steps: S1, constructs a transmission line image-text-label dataset based on transmission line inspection images, which contains image-text-label triplets related to various transmission line components and defects, among which the power semantic knowledge contained in the text data; S2, select the high-performance two-stream vision-language model CN-CLIP as the baseline model, and use the existing general pre-trained representation in it to reduce training costs and improve training efficiency; S3, design a pre-training algorithm unique to the power field, using ITC as the basic pre-training task, while avoiding destroying the original pre-training representation, while introducing power semantic knowledge into the model; S4, based on ITC, CTM is designed as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories; S5, based on ITC, we design DNC as an auxiliary pre-training task to help the model more fully understand the abstract concept of defect-normal, so as to identify defects more effectively; S6, using ITC, CTM and DNC to pre-train a visual basic model for the task of transmission line defect recognition, and using the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage; S7, designed a fine-tuning strategy based on contrastive learning, added ITC on the basis of the downstream transmission line defect recognition task, used the two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and trained the final transmission line defect recognition model; The design of a pre-training algorithm unique to the power field uses ITC as the basic pre-training task, while avoiding destroying the original pre-training representation, and introduces power semantic knowledge into the model, specifically including: First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is: V=f i (I);L=f t (T) (1) Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, respectively. V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively. Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is: Where n is the batch size, τ is the temperature coefficient, and v i and v j represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are the contrastive learning losses from image to text and from text to image, respectively, and l ITC is the total loss of ITC.

2. According to the method for identifying power line defects based on multimodal contrastive learning in claim 1, the transmission line image-text-label dataset is constructed based on the transmission line inspection image, which contains image-text-label triplets related to various transmission line components and defects, specifically including: Firstly, the original transmission line inspection images are annotated and cropped to obtain instance level images of various transmission line components and defects. Each instance level image is accompanied by a corresponding category label. Then, the category labels corresponding to the instance level images are used to fill in the predefined templates or match the predefined descriptions, and a piece of instance level text is generated for each instance level image to complete the construction of the transmission line image-text-label dataset, which contains the power semantic knowledge.

3. According to the method for identifying power line defects based on multimodal contrastive learning in claim 2, the high-performance dual-stream vision-language model CN-CLIP is selected as the baseline model, and the existing general pre-trained representation is used to reduce the training cost and improve the training efficiency, specifically including: Build the CN-CLIP model, and based on the idea of ​​transfer learning, use open source pre-trained weights to initialize the model parameters to avoid the problem of training the model from scratch requiring a large amount of training data and computing resources.

4. According to the method for identifying power line defects based on multimodal contrastive learning in claim 3, the CTM is designed as an auxiliary pre-training task based on ITC to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories, specifically including: The image feature matrix and text feature matrix obtained in ITC are used to perform CTM, which includes four subtasks: image-text type matching ITTM, text-image type matching TITM, image-image type matching IITM, and text-text type matching TTTM. Taking ITTM as an example, first, the text feature matrix and label vector are shuffled: L′,y′=shuffle(L,y) (5) Where shuffle(·) is the shuffle function, y is the label vector composed of category labels, L' and y' are the shuffled text feature matrix and shuffled label vector, respectively, and both share the same random number seed; Secondly, a mapping matrix shared by the four subtasks is used to map the joint feature matrix obtained by concatenating the shuffled text feature matrix and the image feature matrix. After normalization by row, the final matching result matrix is ​​obtained: P ITTM =Softmax(Concat(V,L′)·W CTM ) (6) Where Concat(·,·) is the concatenation function, W CTM is the mapping matrix, P ITTM is the matching result matrix; Again, by comparing the label vector and the shuffled label vector element by element, we determine whether the category relationship between the two elements at the corresponding positions belongs to the same component type and the same state, the same component type but different states, and different types of components, and obtain the label matrix of ITTM: and i ITTM =φ(and i ,and i ′) (7) In the formula, y i ITTM is the relationship label corresponding to the i-th group of samples in the joint feature matrix, y i and i ′ are the category labels of the i-th sample in the label vector and the shuffled label vector, respectively, φ(·,·) is the relationship judgment function; Then, the loss of ITTM is calculated as follows: In the formula, is the jth element of the relationship label corresponding to the i-th group of samples in the joint feature matrix, The jth element in the i-th row of the matching result matrix, l ITTM for the loss of ITTM; Finally, the average of the four CTM subtask losses is taken as the total loss of CTM: l CTM ＝(l ITTM +l TITM +l IITM +l TTTM ) / 4 (9) In the formula, l TITM , l IITM and l TTTM are the losses of TITM, IITM and TTTM tasks respectively, l CTM is the total loss of CTM.

5. According to the method for identifying defects in power transmission lines based on multimodal contrastive learning in claim 4, the design of DNC as an auxiliary pre-training task based on ITC helps the model to more fully understand the abstract concept of defect-normal, thereby more effectively identifying defects, specifically comprising: The DNC is performed using the image feature matrix obtained in ITC. The number of subtasks included is determined according to the number of component types with both defective and normal categories in the transmission line image-text-label dataset. Taking the grading ring comparison GRD subtask as an example, first, according to the label vector and the category index of the defective and normal grading rings, the feature vectors corresponding to the defective grading ring and the normal grading ring samples are screened out from the image feature matrix: Where σ(·,·,·) is the screening function, i DGR and i NGR are the category indexes of defective and normal grading rings, V DGR and V NGR are the defective balancing ring feature matrix and the normal balancing ring feature matrix composed of the feature vectors corresponding to the defective balancing ring and the normal balancing ring samples, respectively. and They represent the feature vector of the i-th defective balancing ring and the feature vector of the j-th normal balancing ring, respectively. k and q are the numbers of defective balancing rings and normal balancing rings in the training batch, respectively. Secondly, the defective pressure equalizing ring feature matrix and the normal pressure equalizing ring feature matrix are concatenated to obtain the defective-normal pressure equalizing ring feature matrix containing all the pressure equalizing ring features, and the cosine similarity between the defective-normal pressure equalizing ring feature matrix and itself is calculated. After normalization, the similarity score matrix can be obtained: In GR =Concat(V DGR ,V NGR ) (12) Where V GR represents the defect-normal grading ring characteristic matrix, S GR is the similarity score matrix, represents the similarity score between the i-th pressure equalizing ring and the j-th pressure equalizing ring, and m is the total number of pressure equalizing rings in the training batch; Again, the label matrix of GRC is generated according to the number of defective equalizing rings and normal equalizing rings in the training batch. The formula is: In the formula, Z GR is the label matrix of GRC, is the similarity label between the i-th pressure equalizing ring and the j-th pressure equalizing ring; Then, the loss of GRC is calculated as follows: In the formula, l GRC for the loss of GRC; Finally, the average of all DNC subtask losses is taken as the total loss of DNC: In the formula, l DNC(i) represents the loss of the ith subtask of DNC, t is the number of subtasks of DNC, l DNC is the total loss of DNC.

6. According to the method for identifying power line defects based on multimodal contrastive learning in claim 5, the method uses ITC, CTM and DNC to pre-train a visual basic model for the task of identifying power line defects, and uses the image encoder therein as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage, specifically including: First, the three pre-training tasks mentioned above are used to perform pre-training on CN-CLIP specifically for the power field, and a visual basic model for the task of transmission line defect recognition is obtained. The total loss calculation formula in the pre-training stage is: l pre-train =λ1l ITC +λ2l CTM +λ3l DNC (18) In the formula, l pre-train is the total loss in the pre-training stage, and λ1, λ2, and λ3 are used to adjust the coefficients of the proportion of the three losses in the total loss; Then, the image encoder in the visual base model for the power transmission line defect recognition task is used as the backbone network of the downstream defect recognition model, connected to the task-specific head network for performing the defect recognition task, and the downstream defect recognition model is constructed.

7. According to claim 6, a transmission line defect recognition method based on multimodal contrastive learning, the design is based on a fine-tuning strategy of contrastive learning, ITC is added on the basis of the downstream transmission line defect recognition task, and the entire recognition model is fine-tuned using two tasks to alleviate the overfitting problem in the fine-tuning stage, and the final transmission line defect recognition model is trained, which specifically includes: When the downstream transmission line defect recognition task is a defect classification task, first, the image feature matrix output by the image encoder is passed into the task-specific head network to obtain the defect classification task output. The corresponding classification loss calculation formula is: In the formula, is the label of the i-th sample belonging to the j-th category, is the model's predicted probability that the i-th sample belongs to the j-th category, c is the category of the defect classification task, and l CLS is the classification loss; Secondly, fill the predefined template or match the predefined description with the category label of the downstream defect classification dataset, generate a corresponding text for each image, pass all the text into the text encoder to obtain the text feature matrix, and calculate the ITC loss according to formula (2)-formula (4). The total loss in the fine-tuning stage is the sum of the losses of the two tasks, and the formula is: l fine-tune ＝l CLS +l ITC (20) In the formula, l fine-tune is the total loss in the fine-tuning phase; When the downstream transmission line defect recognition task is a defect detection task, first, the instances in the downstream defect detection dataset are cropped into instance level images of various scales according to the annotation boxes. Then, the backbone network of the defect detection model and the classification branch of the task-specific head network are fine-tuned according to the method of the downstream transmission line defect recognition task as a defect classification task. The loss at this stage is calculated as: In the formula, is the fine-tuning loss of the first stage; Then, ITC is removed and the entire defect detection model is fine-tuned using two tasks, defect classification and bounding box regression, on the original downstream defect detection dataset. The loss at this stage is calculated as: In the formula, is the fine-tuning loss of the second stage, l BBOX is the bounding box regression loss; Finally, the text encoder is removed to obtain the final transmission line defect recognition model. During model inference, the image encoder and the task-specific head network are used to infer the test image to obtain the transmission line defect recognition output, which is: Output=Taskhead(f i (x)) (23) Where x is the test image input, Output is the transmission line defect recognition output, and Taskhead(·) is the task-specific head network.

8. An electronic device, characterized in that: The electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement a transmission line defect identification method based on multimodal contrast learning as described in any one of claims 1-7.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and the computer executable instructions are used to execute a method for identifying power transmission line defects based on multimodal contrast learning as described in any one of claims 1 to 7.

Images