A method for transmission line defect identification based on multimodal contrastive learning

Abstract

This paper provides a transmission line defect recognition method based on multimodal contrastive learning. This method constructs a transmission line image-text-label dataset. While leveraging the powerful pre-trained representations in existing visual-language models, it also incorporates the power semantics inherent in the textual modal data through auxiliary pre-training tasks unique to the power sector, effectively improving the accuracy of transmission line defect recognition. Furthermore, a fine-tuning strategy based on contrastive learning is designed. It adds ITC to the downstream transmission line defect recognition task. These two tasks fine-tune the overall recognition model to mitigate overfitting during the fine-tuning phase, ultimately yielding the final transmission line defect recognition model.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method for identifying defects in power transmission lines based on multimodal contrast learning. Background Art

[0002] The stable operation of transmission lines is closely related to maintaining the safety of the power system and ensuring the quality of power supply. Due to long-term exposure to harsh outdoor environments and the combined effects of various external factors such as wind, rain, snow, and lightning, the hardware in the transmission lines is prone to defects and failures of varying degrees, posing a serious threat to the safety of the power system. Due to the long distances of transmission lines and the complex terrain conditions in which some towers are located, traditional manual inspection methods require a lot of manpower costs and are inefficient, and can no longer meet the needs of new power system construction. With the rapid development of drone-based transmission line inspection technology in recent years, current transmission line defect identification methods mainly use deep learning-based computer vision technology to process and analyze aerial images of transmission lines. This inspection model effectively improves the intelligence level of power grid operation and maintenance, and effectively reduces the burden of transmission line inspection operations.

[0003] However, the existing transmission line defect recognition model based on deep learning mainly uses the traditional supervised pre-training model for transfer learning. Constrained by the accessible categories and the scale of pre-training data, the generalization ability of the traditional supervised pre-training model is relatively limited, and it lacks the power semantic knowledge unique to the power field. The research on the visual-language model, which has received widespread attention in recent years, provides ideas for overcoming the above problems. Due to pre-training on massive image-text data at the network level, the visual-language model has stronger generalization capabilities than the traditional supervised pre-training model, providing a better training starting point for training the transmission line defect recognition model based on deep learning. However, due to the existence of data barriers in the power industry, the existing visual-language model still only has access to limited power-related data during the pre-training stage and lacks power semantic knowledge. There are still two problems in applying it to the task of transmission line defect recognition:

[0004] 1. Existing vision-language models lack inherent knowledge of power semantics, and their corresponding pre-training methods fail to fully account for the unique inter-class relationships within transmission line inspection data. Consequently, directly fine-tuning existing vision-language models on transmission line defect recognition data yields poor results. Furthermore, even if existing vision-language models are further pre-trained on power data with domain-specific knowledge and then fine-tuned, only suboptimal results are achieved.

[0005] 2. Due to limited data in the power sector, existing vision-language models inevitably suffer from overfitting after domain-specific pre-training. Fine-tuning the model using supervised learning objectives for the transmission line defect recognition task further exacerbates the overfitting problem, thereby limiting the model's accuracy in this task.

[0006] Therefore, in this context, the high-performance two-stream vision-language model CN-CLIP is applied to the task of transmission line defect recognition. Based on the basic characteristics of transmission line inspection data, the problems existing in the original vision-language model in transmission line application scenarios are improved, and a transmission line defect recognition model with higher accuracy is trained, which has important research significance for the intelligent inspection of transmission lines. Summary of the Invention

[0007] The purpose of this invention is to design a method for power line defect recognition based on multimodal contrastive learning. This method overcomes the negative impacts on recognition accuracy caused by existing visual-language models' lack of power semantic knowledge, visual-language model pre-training methods that ignore inter-class relationships within transmission line inspection data, and increased overfitting during visual-language model fine-tuning. This invention employs two domain-specific auxiliary pre-training tasks to help the model better distinguish between different semantic categories and understand the abstract concept of defect versus normal. Furthermore, a fine-tuning strategy based on contrastive learning effectively mitigates overfitting during the fine-tuning phase, thereby improving the accuracy of power line defect recognition.

[0008] To achieve the above object, the present invention provides the following solutions:

[0009] A method for identifying defects in power transmission lines based on multimodal contrastive learning, the method comprising the following steps:

[0010] S1, based on the transmission line inspection images, builds a transmission line image-text-label dataset, which contains image-text-label triplets related to various transmission line components and defects, including the power semantic knowledge contained in the text data;

[0011] S2, selects the high-performance two-stream vision-language model CN-CLIP as the baseline model, and uses the existing general pre-trained representations in it to reduce training costs and improve training efficiency;

[0012] S3, design a pre-training algorithm specific to the power sector, using IT C as the basic pre-training task, while avoiding destroying the original pre-training representation and introducing power semantic knowledge into the model;

[0013] S4, based on ITC, CTM is designed as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0014] S5, based on ITC, DNC is designed as an auxiliary pre-training task to help the model more fully understand the abstract concept of defect-normal, thereby more effectively identifying defects;

[0015] S6 uses ITC, CTM, and DNC to pre-train a visual base model for transmission line defect recognition tasks, and uses the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage;

[0016] S7, designed a fine-tuning strategy based on contrastive learning. It added ITC to the downstream transmission line defect recognition task. Using these two tasks, the entire recognition model was fine-tuned to alleviate the overfitting problem during the fine-tuning phase. The final transmission line defect recognition model was trained.

[0017] The design of a pre-training algorithm specific to the power sector uses IT C as the basic pre-training task. While avoiding destroying the original pre-training representation, it introduces power semantic knowledge into the model. Specifically, it includes:

[0018] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0019] V=f i (I); L = f t (T) (1)

[0020] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively;

[0021] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0022]

[0023] Where n is the batch size, τ is the temperature coefficient, and v i and v j Represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are contrastive learning losses from image to text and from text to image, respectively, l ITC is the total loss of ITC.

[0024] An electronic device comprises: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the above-mentioned transmission line defect identification method based on multimodal contrast learning.

[0025] A storage medium, characterized in that the storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned transmission line defect identification method based on multimodal contrast learning.

[0026] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] The embodiment of the present invention discloses a method for identifying power line defects based on multimodal contrastive learning, comprising the following steps: constructing a transmission line image-text-label dataset, comprising image-text-label triplets related to various transmission line components and defects, for pre-training a visual base model for the task of identifying transmission line defects; selecting a two-stream vision-language model CN-CLIP as a baseline model; designing a pre-training algorithm specific to the power field, taking image-text contrast learning (ITC) as a basic pre-training task, and introducing power semantic knowledge contained in the transmission line image-text-label dataset into the model without destroying the original pre-training representation; designing two auxiliary pre-training tasks, component type matching (CTM) and defect-normality comparison (DNC), based on the ITC, and making full use of the unique inter-class relationship within the transmission line inspection data to help the model more effectively distinguish the semantics of different categories and more fully understand the abstract concept of defect-normal; using ITC to identify the power line defects based on the image-text contrastive learning (ITC) algorithm; and finally, designing a method for identifying power line defects based on the image-text contrastive learning (ITC) algorithm. TC, CTM, and DNC pre-train a visual foundation model for the transmission line defect recognition task. The image encoder in this model serves as the backbone network for the downstream recognition model, transferring the rich power semantic knowledge learned during the pre-training phase. A fine-tuning strategy based on contrastive learning is designed, adding ITC to the downstream transmission line defect recognition task. These two tasks fine-tune the entire recognition model to mitigate overfitting during the fine-tuning phase, ultimately training the final transmission line defect recognition model. This paper applies the CN-CLIP algorithm to transmission line defect recognition. By introducing the power semantic knowledge contained in text modal data, the accuracy of transmission line defect recognition can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly demonstrate the embodiments of the present invention or the technical solutions in the prior art, the drawings involved in the embodiments are briefly described below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of a method for identifying transmission line defects based on multimodal contrast learning according to an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of the execution of the CTM pre-training task according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the execution of the DNC pre-training task according to an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the overall training process of an embodiment of the present invention;

[0033] Figure 5 A heat map showing the classification of transmission line defects according to an embodiment of the present invention;

[0034] Figure 6 This is a diagram showing the effect of transmission line defect detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] The purpose of the present invention is to provide a transmission line defect recognition method based on multimodal contrastive learning, which solves the problems of the lack of power semantic knowledge in the existing visual-language model, the neglect of the internal inter-class relationship of the transmission line inspection data by the visual-language model pre-training method, and the aggravation of overfitting in the process of fine-tuning the visual-language model, so as to achieve the purpose of improving the accuracy of transmission line defect recognition.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] like Figure 1As shown in FIG, a flow chart of a method for identifying defects in a transmission line based on multimodal contrastive learning provided by the present invention comprises the following steps:

[0039] S1, based on the transmission line inspection images, builds a transmission line image-text-label dataset, which contains image-text-label triplets related to various transmission line components and defects, including the power semantic knowledge contained in the text data;

[0040] S2, selects the high-performance two-stream vision-language model CN-CL IP as the baseline model, and uses the existing general pre-trained representations in it to reduce training costs and improve training efficiency;

[0041] S3, design a pre-training algorithm specific to the power sector, using IT C as the basic pre-training task, while avoiding destroying the original pre-training representation and introducing power semantic knowledge into the model;

[0042] S4, based on ITC, CTM is designed as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0043] S5, based on ITC, DNC is designed as an auxiliary pre-training task to help the model more fully understand the abstract concept of defect-normal, thereby more effectively identifying defects;

[0044] S6 uses ITC, CTM, and DNC to pre-train a visual base model for transmission line defect recognition tasks, and uses the image encoder as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage;

[0045] S7 designs a fine-tuning strategy based on contrastive learning. It adds ITC to the downstream transmission line defect recognition task and uses the two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage. The final transmission line defect recognition model is trained.

[0046] Traditional supervised pre-training methods typically use image-labeled data containing limited concepts and image classification as the pre-training task to pre-train the model. The resulting models have limited generalization capabilities and lack power semantic knowledge. Therefore, they only achieve suboptimal results when transferred to transmission line inspection data. Multimodal pre-training, on the other hand, utilizes a large amount of image-text data, typically using ITC as the pre-training task. This gives the model strong generalization capabilities while also explicitly incorporating the power semantic knowledge contained in text modal data. Therefore, it offers advantages over traditional supervised pre-training methods in the power sector. However, multimodal pre-training requires a large number of instance-level images and instance-level text samples as support. Therefore, it is necessary to crop each component and defect instance from the original inspection image containing multiple component and defect instances into a separate instance-level image and generate corresponding instance-level text. Furthermore, due to the requirements of pre-training algorithms specific to the power sector, it is also necessary to retain the category label corresponding to each instance-level image. Therefore, this method uses image-text-labeled data to pre-train the model. Therefore, in step S1, a transmission line image-text-label dataset is constructed based on the transmission line inspection images, which includes image-text-label triplets related to various transmission line components and defects, specifically including:

[0047] First, the original transmission line inspection images are labeled and cropped to obtain instance level images of various transmission line components and defects. Each instance level image is accompanied by a corresponding category label.

[0048] Then, the category labels corresponding to the instance level images are used to fill in the predefined templates or match the predefined descriptions, and a piece of instance level text is generated for each instance level image, completing the construction of the transmission line image-text-label dataset.

[0049] In this method, the original CN-CLIP model does not contain power semantic knowledge, and training a visual-language model from scratch requires billions of data and a large amount of computing power, which is difficult to meet with the available transmission line inspection data and computing resources. Therefore, in step S2, the high-performance two-stream visual-language model CN-CLIP is selected as the baseline model, and the existing general pre-trained representations in it are used to reduce training costs and improve training efficiency. Specifically, the following steps are included:

[0050] Build the CN-CLIP model and, based on the idea of ​​transfer learning, initialize the model parameters with open-source pre-trained weights to avoid the problem of requiring a large amount of training data and computing power resources to train the model from scratch.

[0051] In this method, the original CN-CLIP model is pre-trained using the ITC task on massive, web-scale image-text data. Therefore, using only other pre-training tasks in the power sector-specific pre-training would damage the original pre-trained representation. Therefore, in step S3, a pre-training algorithm specific to the power sector is designed, using ITC as the fundamental pre-training task. While avoiding damage to the original pre-trained representation, power semantic knowledge is introduced into the model. Specifically, the algorithm includes:

[0052] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0053] V=f i (I); L = f t (T) (1)

[0054] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively.

[0055] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0056]

[0057] Where n is the batch size, τ is the temperature coefficient, and v i and v j Represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are contrastive learning losses from image to text and from text to image, respectively, l ITC is the total loss of ITC.

[0058] In this method, considering that there are two attributes, component type and component status, implicitly in the transmission line inspection data, the transmission line inspection data has a more complex category relationship than the general domain data. Specifically, the category relationship between two instances in the general domain data can be summarized as "same" and "different", while the category relationship between two instances in the transmission line inspection data has three cases, namely: "same component type and same status", "same component type but different status" and "different type components". For example: the category relationship between two normal equalizing ring instances is "same component type and same status"; the category relationship between a normal equalizing ring instance and a defective equalizing ring instance is "same component type but different status"; and the category relationship between any equalizing ring instance and any shielding ring instance is "different type components". Distinguishing the above three category relationships by training the model helps to better distinguish categories with similar semantics. Therefore, in the step S4, CTM is designed as an auxiliary pre-training task based on ITC to help the model more effectively distinguish the semantics of different component types and component statuses to avoid confusion of similar categories, specifically including:

[0059] CTM uses the image and text feature matrices obtained from ITC. This involves four subtasks: image-text type matching (ITTM), text-image type matching (TITM), image-image type matching (IITM), and text-text type matching (TTTM). Taking ITTM as an example, first, the text feature matrix and label vector are shuffled:

[0060] L′,y′=shuffle(L,y) (5)

[0061] Where shuffle(·) is the shuffle function, y is the label vector consisting of category labels, L' and y' are the shuffled text feature matrix and shuffled label vector, respectively, and both share the same random number seed.

[0062] Secondly, a mapping matrix shared by the four subtasks is used to map the joint feature matrix obtained by concatenating the scrambled text feature matrix and the image feature matrix. After normalization by row, the final matching result matrix is ​​obtained:

[0063] P ITTM =Softmax(Concat(V,L′)·W CTM ) (6)

[0064] Where Concat(·,·) is the concatenation function, W CTM is the mapping matrix, P ITTM is the matching result matrix.

[0065] Again, by comparing the label vector and the scrambled label vector element by element, we determine whether the category relationship between the two elements at the corresponding position belongs to the same component type and the same state, the same component type but different states, or different types of components, and obtain the ITTM label matrix:

[0066] y i ITTM =φ(y i ,y i ′) (7)

[0067] Where y i ITTM is the relationship label corresponding to the i-th group of samples in the joint feature matrix, y i and y i ′ are the category labels of the i-th sample in the label vector and the shuffled label vector, respectively, and φ(·,·) is the relationship judgment function.

[0068] Then, calculate the loss of ITTM as follows:

[0069]

[0070] Where, is the jth element of the relationship label corresponding to the i-th group of samples in the joint feature matrix, The jth element in the i-th row of the matching result matrix, l ITTM For ITTM's loss.

[0071] Finally, the average of the four CTM subtask losses is taken as the total loss of CTM:

[0072] l CTM =(l ITTM +l TITM +l IITM +l TTTM ) / 4 (9)

[0073] Where, l TITM 、l IITM and l TTTM are the losses of TITM, IITM and TTTM tasks respectively, l CTM is the total loss of CTM. The execution process of the four CTM subtasks is as follows Figure 2 As shown, by learning four CTM subtasks, the model can align the semantic information about various parts and their states in the visual and language modalities, thereby better distinguishing different categories.

[0074] In this method, it is considered that understanding the abstract concept of defect-normal is of vital importance to the downstream defect identification task, and the traditional pre-training tasks have ignored the implicit component status attributes in the transmission line inspection data, and therefore cannot help the model learn the abstract concept of defect-normal. This method uses the characteristics that the defect and normal categories under each type of component in the transmission line inspection data exist in pairs or groups (such as: the equalizing ring component type contains two categories of "normal equalizing ring" and "broken equalizing ring"). By comparing the defect and normal category samples under the same component type, the model can better recognize the defect-normal concept in this type of component. Therefore, in the step S5, DNC is designed as an auxiliary pre-training task based on I TC to help the model more fully understand the abstract concept of defect-normal, so as to more effectively identify defects, specifically including:

[0075] The image feature matrix obtained from ITC is used to perform DNC. The number of subtasks included is determined by the number of component types with both defective and normal categories in the transmission line image-text-label dataset. Taking the grading ring contrast (GRD) subtask as an example, its execution process is as follows: Figure 3 First, based on the label vector and the category index of the defective and normal equalizing rings, the feature vectors corresponding to the defective and normal equalizing ring samples are filtered out from the image feature matrix:

[0076]

[0077] Where σ(·,·,·) is the screening function, i DGR and i NGR are the category indexes of defective and normal grading rings, V DGR and V NGR The defective grading ring feature matrix and the normal grading ring feature matrix are composed of the feature vectors corresponding to the defective grading ring and the normal grading ring samples, respectively. and They represent the feature vector of the i-th defective balancing ring and the feature vector of the j-th normal balancing ring, respectively. k and q are the number of defective balancing rings and normal balancing rings in the training batch, respectively.

[0078] Secondly, the defective equalizing ring feature matrix and the normal equalizing ring feature matrix are concatenated to obtain the defective-normal equalizing ring feature matrix containing all the equalizing ring features. The cosine similarity between the defective-normal equalizing ring feature matrix and itself is calculated, and after normalization, the similarity score matrix can be obtained:

[0079] V GR =Concat(V DGR ,VNGR ) (12)

[0080]

[0081] Where V GR Denotes the defect-normal grading ring characteristic matrix, S GR is the similarity score matrix, represents the similarity score between the i-th pressure equalizing ring and the j-th pressure equalizing ring, and m is the total number of pressure equalizing rings in the training batch.

[0082] Again, the GRC label matrix is ​​generated according to the number of defective balancing rings and normal balancing rings in the training batch. The formula is:

[0083]

[0084] Where Z GR is the label matrix of GRC, is the similarity label between the i-th pressure equalizing ring and the j-th pressure equalizing ring.

[0085] Then, calculate the loss of GRC as follows:

[0086]

[0087] Where, l GRC For the loss of GRC.

[0088] Finally, the average of all DNC subtask losses is taken as the total loss of DNC:

[0089]

[0090] Where, l DNC(i) represents the loss of the ith subtask of DNC, t is the number of subtasks of DNC, l DNC is the total loss of DNC.

[0091] The present invention integrates the three pre-training tasks mentioned in steps S3, S4 and S5 into the power field-specific pre-training of the baseline model CN-CLIP mentioned in step S2, forming a complete set of power field-specific pre-training algorithms, such as Figure 4 Shown on the left.

[0092] In step S6, a visual basic model for the power transmission line defect recognition task is pre-trained using IT CTM and DNC, and the image encoder therein is used as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training stage, specifically including:

[0093] First, we use the three pre-training tasks mentioned above to pre-train CN-CLIP specifically for the power sector, obtaining a visual base model for the task of transmission line defect recognition. The total loss calculation formula for the pre-training stage is:

[0094] l pre-train =λ1l ITC +λ2l CTM +λ3l DNC (18)

[0095] Where, l pre-train is the total loss in the pre-training stage, and λ1, λ2, and λ3 are used to adjust the coefficients of the proportion of the three losses in the total loss.

[0096] Then, the image encoder in the visual base model for the transmission line defect recognition task is used as the backbone network of the downstream defect recognition model, connected to the task-specific head network for performing the defect recognition task, and the downstream defect recognition model is constructed.

[0097] In this method, given the limited scale of transmission line inspection data, the pre-trained visual base model for the transmission line defect recognition task inevitably suffers from overfitting. Using a supervised learning objective with a strong supervisory signal to guide the model fine-tuning process according to the traditional fine-tuning paradigm will further exacerbate overfitting. However, the weak supervisory signal provided by the contrastive learning objective used in pre-training can, to a certain extent, mitigate overfitting. Therefore, in step S7, a fine-tuning strategy based on contrastive learning is designed. It adds ITC to the downstream transmission line defect recognition task. These two tasks are used to fine-tune the entire recognition model to alleviate the overfitting problem during the fine-tuning phase. The final transmission line defect recognition model is trained, specifically including:

[0098] When the downstream transmission line defect recognition task is a defect classification task, first, the image feature matrix output by the image encoder is passed to the task-specific head network to obtain the defect classification task output. The corresponding classification loss calculation formula is:

[0099]

[0100] Where, is the label that the i-th sample belongs to the j-th category, is the model's predicted probability that the i-th sample belongs to the j-th category, c is the category of the defect classification task, l CLS is the classification loss.

[0101] Secondly, fill the predefined template or match the predefined description with the category label of the downstream defect classification dataset, generate a corresponding text for each image, pass all the texts into the text encoder to obtain the text feature matrix, and calculate the ITC loss according to formula (2)-formula (4). The total loss in the fine-tuning stage is the sum of the two task losses, which is:

[0102] l fine-tune =l CLS +l ITC (20)

[0103] Where, l fine-tune is the total loss in the fine-tuning stage.

[0104] When the downstream transmission line defect recognition task is a defect detection task, first, the instances in the downstream defect detection dataset are cropped into instance-level images of various scales according to the annotation boxes. Then, the backbone network and the classification branch of the task-specific head network of the defect detection model are fine-tuned using the method based on the case where the downstream transmission line defect recognition task is a defect classification task. The loss at this stage is calculated as:

[0105]

[0106] Where, is the fine-tuning loss of the first stage.

[0107] Then, ITC is removed and the entire defect detection model is fine-tuned using two tasks, defect classification and bounding box regression, on the original downstream defect detection dataset. The loss at this stage is calculated as:

[0108]

[0109] Where, is the fine-tuning loss of the second stage, l BBOX is the bounding box regression loss.

[0110] Finally, remove the text encoder to obtain the final transmission line defect recognition model. The fine-tuning strategy can be used Figure 4 The right side summarizes the representation. During model inference, the image encoder and task-specific head network are used to infer the test image and obtain the transmission line defect recognition output, which is formulated as:

[0111] Output=Taskhead(f i (x)) (23)

[0112] Where x is the test image input, Output is the transmission line defect recognition output, and Taskhead(·) is the task-specific head network.

[0113] The thermal map and detection effect generated by the method of the present invention in the task of identifying transmission line defects are as follows: Figure 5 and Figure 6 As shown. Based on the dual-stream vision-language model CN-CL IP, the present invention designs a pre-training algorithm specific to the power sector. It uses ITC as the basic pre-training task to prevent damage to the original pre-training representation. Based on ITC, CTM is designed as an auxiliary pre-training task. By training the model to distinguish the category relationship between any two instances in transmission line inspection data, the model can more effectively distinguish the semantics of different categories. Based on ITC, DNC is designed as another auxiliary pre-training task. By training the model to compare defective and normal category samples of the same component type, the model can more fully understand the abstract concept of defective-normal. To address the overfitting problem that occurs during the fine-tuning phase of the pre-trained visual base model for the transmission line defect recognition task, a fine-tuning strategy based on contrastive learning is designed. ITC is added to the downstream transmission line defect recognition task, and a contrastive learning objective with a weak supervisory signal is used to alleviate the overfitting problem during the fine-tuning phase. Thus, the method described in the present invention can help the transmission line defect recognition model learn transmission line-related semantic information while understanding the abstract visual concept of normal-defective. The present invention effectively introduces the power semantic knowledge contained in text modal data into the transmission line defect recognition model, thereby effectively improving the recognition accuracy of transmission line defects.

[0114] At the same time, the present application also provides a device corresponding to the method for identifying transmission line defects based on multimodal contrastive learning, as well as an electronic device and a storage medium, as follows:

[0115] The device comprises:

[0116] The S1 module is used to construct a transmission line image-text-label dataset based on transmission line inspection images. It contains image-text-label triplets related to various transmission line components and defects, including the power semantic knowledge contained in the text data.

[0117] The S2 module selects the high-performance two-stream vision-language model CN-CLIP as the baseline model and uses the existing general pre-trained representations in it to reduce training costs and improve training efficiency;

[0118] The S3 module is used to design a pre-training algorithm specific to the power sector. It uses IT C as the basic pre-training task and introduces power semantic knowledge into the model while avoiding destroying the original pre-training representation.

[0119] The S4 module is used to design CTM as an auxiliary pre-training task based on ITC, helping the model to more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories;

[0120] The S5 module is used to design DNC as an auxiliary pre-training task based on ITC, helping the model to more fully understand the abstract concept of defect-normality and thus more effectively identify defects;

[0121] The S6 module is used to pre-train a basic visual model for transmission line defect recognition using ITC, CTM, and DNC. The image encoder in the model serves as the backbone network of the downstream recognition model, transferring the rich power semantic knowledge learned in the pre-training phase.

[0122] The S7 module is used to design a fine-tuning strategy based on contrastive learning. It adds ITC to the downstream transmission line defect recognition task. It uses these two tasks to fine-tune the entire recognition model to alleviate the overfitting problem in the fine-tuning stage, and then trains the final transmission line defect recognition model.

[0123] The S3 module is specifically used to:

[0124] First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is:

[0125] V=f i (I); L = f t (T) (1)

[0126] Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively;

[0127] Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is:

[0128]

[0129] Where n is the batch size, τ is the temperature coefficient, and v i and v j Represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are contrastive learning losses from image to text and from text to image, respectively, l ITC is the total loss of ITC.

[0130] An electronic device comprises: at least one memory and at least one processor; the memory stores a program, the processor calls the program stored in the memory, and the program is used to implement the transmission line defect identification method based on multimodal contrast learning.

[0131] A storage medium stores computer-executable instructions for executing a method for identifying power transmission line defects based on multimodal contrast learning.

[0132] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for identifying transmission line defects based on multimodal contrastive learning, characterized in that: The method comprises the following steps: S1, based on the transmission line inspection images, builds a transmission line image-text-label dataset, which contains image-text-label triplets related to various transmission line components and defects, including the power semantic knowledge contained in the text data; S2, selects the high-performance two-stream vision-language model CN-CLIP as the baseline model, and uses the existing general pre-trained representations in it to reduce training costs and improve training efficiency; S3: Design a pre-training algorithm specific to the power sector, using ITC as the basic pre-training task. This algorithm introduces power semantic knowledge into the model while avoiding destroying the original pre-training representation. S4, based on ITC, we design CTM as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories; S5, based on ITC, we design DNC as an auxiliary pre-training task to help the model more fully understand the abstract concept of defect-normal, thereby more effectively identifying defects; S6 uses ITC, CTM, and DNC to pre-train a visual base model for transmission line defect recognition tasks. The image encoder in the model is used as the backbone network of the downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training phase. S7, designed a fine-tuning strategy based on contrastive learning. It added ITC to the downstream transmission line defect recognition task. Using these two tasks, the entire recognition model was fine-tuned to alleviate the overfitting problem during the fine-tuning phase. The final transmission line defect recognition model was trained. The pre-training algorithm designed for the power sector uses ITC as the basic pre-training task. While avoiding destroying the original pre-training representation, it introduces power semantic knowledge into the model. Specifically, it includes: First, the image encoder and text encoder of the CN-CLIP model are used to extract the features of the input image and text respectively, and the image feature matrix and text feature matrix are obtained. The formula is: V=f i (I);L=f t (T) (1) Where I and T represent the image and text input to CN-CLIP respectively, and f i (·) and f t (·) are the image encoder and text encoder in CN-CLIP, V and L are the image feature matrix and text feature matrix output by CN-CLIP, respectively; Then, the cosine similarity between the image feature matrix and the text feature matrix is ​​calculated, and the ITC loss is calculated from two directions. The formula is: Where n is the batch size, τ is the temperature coefficient, and v i and v j Represents the feature vector of the i-th and j-th samples in the image feature matrix, l i and l j Represents the feature vectors of the i-th and j-th samples in the text feature matrix, and are contrastive learning losses from image to text and from text to image, respectively, l ITC is the total loss of ITC.

2. According to the method for identifying power line defects based on multimodal contrastive learning in claim 1, the transmission line image-text-label dataset is constructed based on the transmission line inspection images, and contains image-text-label triplets related to various transmission line components and defects, specifically including: First, the original transmission line inspection images are annotated and cropped to obtain instance-level images of various transmission line components and defects. Each instance-level image is accompanied by a corresponding category label. Then, the category labels corresponding to the instance-level images are used to fill in predefined templates or match predefined descriptions, and a piece of instance-level text is generated for each instance-level image to complete the construction of the transmission line image-text-label dataset, which contains the power semantic knowledge.

3. According to the method for identifying power line defects based on multimodal contrastive learning in claim 2, the method selects the high-performance two-stream vision-language model CN-CLIP as the baseline model and utilizes the existing general pre-trained representations therein to reduce training costs and improve training efficiency, specifically comprising: Build the CN-CLIP model and, based on the idea of ​​transfer learning, initialize the model parameters with open-source pre-trained weights to avoid the problem of requiring a large amount of training data and computing power resources to train the model from scratch.

4. The method for identifying power line defects based on multimodal contrastive learning according to claim 3, wherein the CTM is designed based on ITC as an auxiliary pre-training task to help the model more effectively distinguish the semantics of different component types and component states to avoid confusion between similar categories, specifically comprising: The image feature matrix and text feature matrix obtained in ITC are used to perform CTM, which includes four subtasks: image-text type matching ITTM, text-image type matching TITM, image-image type matching IITM, and text-text type matching TTTM. Taking ITTM as an example, first, the text feature matrix and label vector are shuffled: L′,y′=shuffle(L,y) (5) Where shuffle(·) is the shuffle function, y is the label vector composed of category labels, L' and y' are the shuffled text feature matrix and shuffled label vector, respectively, and both share the same random number seed; Secondly, a mapping matrix shared by the four subtasks is used to map the joint feature matrix obtained by concatenating the scrambled text feature matrix and the image feature matrix. After normalization by row, the final matching result matrix is ​​obtained: P ITTM =Softmax(Concat(V,L′)·W CTM ) (6) Where Concat(·,·) is the concatenation function, W CTM is the mapping matrix, P ITTM is the matching result matrix; Again, by comparing the label vector and the scrambled label vector element by element, we determine whether the category relationship between the two elements at the corresponding position belongs to the same component type and the same state, the same component type but different states, or different types of components, and obtain the ITTM label matrix: and i ITTM =φ(and i ,and i ′) (7) Where y i ITTM is the relationship label corresponding to the i-th group of samples in the joint feature matrix, y i and y i ′ are the category labels of the i-th sample in the label vector and the shuffled label vector, respectively, and φ(·,·) is the relationship judgment function; Then, calculate the loss of ITTM as follows: Where, is the jth element of the relationship label corresponding to the i-th group of samples in the joint feature matrix, The jth element in the i-th row of the matching result matrix, l ITTM for ITTM's losses; Finally, the average of the four CTM subtask losses is taken as the total loss of CTM: l CTM ＝(l ITTM +l TITM +l IITM +l TTTM ) / 4 (9) Where, l TITM 、l IITM and l TTTM are the losses of TITM, IITM and TTTM tasks respectively, l CTM is the total loss of CTM.

5. The method for identifying power line defects based on multimodal contrastive learning according to claim 4, wherein the DNC is designed as an auxiliary pre-training task based on ITC to help the model more fully understand the abstract concept of defect-normality, thereby more effectively identifying defects, specifically comprising: The DNC is performed using the image feature matrix obtained from ITC. The number of subtasks included is determined by the number of component types with both defective and normal categories in the transmission line image-text-label dataset. Taking the grading ring comparison GRD subtask as an example, first, based on the label vector and the category indexes of the defective and normal grading rings, the feature vectors corresponding to the defective and normal grading ring samples are filtered out from the image feature matrix: Where σ(·,·,·) is the screening function, i DGR and i NGR are the category indexes of defective and normal grading rings, V DGR and V NGR The defective grading ring feature matrix and the normal grading ring feature matrix are composed of the feature vectors corresponding to the defective grading ring and the normal grading ring samples, respectively. and They represent the feature vector of the i-th defective balancing ring and the feature vector of the j-th normal balancing ring, respectively. k and q are the numbers of defective balancing rings and normal balancing rings in the training batch, respectively. Secondly, the defective equalizing ring feature matrix and the normal equalizing ring feature matrix are concatenated to obtain the defective-normal equalizing ring feature matrix containing all the equalizing ring features. The cosine similarity between the defective-normal equalizing ring feature matrix and itself is calculated, and after normalization, the similarity score matrix can be obtained: In GR =Concat(V DGR ,V NGR ) (12) Where V GR Denotes the defect-normal grading ring characteristic matrix, S GR is the similarity score matrix, represents the similarity score between the i-th pressure equalizing ring and the j-th pressure equalizing ring, and m is the total number of pressure equalizing rings in the training batch; Again, the GRC label matrix is ​​generated according to the number of defective balancing rings and normal balancing rings in the training batch. The formula is: Where Z GR is the label matrix of GRC, is the similarity label between the i-th pressure equalizing ring and the j-th pressure equalizing ring; Then, calculate the loss of GRC as follows: Where, l GRC for GRC's losses; Finally, the average of all DNC subtask losses is taken as the total loss of DNC: Where, l DNC(i) represents the loss of the ith subtask of DNC, t is the number of subtasks of DNC, l DNC is the total loss of DNC.

6. The method for identifying power line defects based on multimodal contrastive learning according to claim 5, wherein the method pre-trains a visual base model for the task of identifying power line defects using ITC, CTM, and DNC, and uses the image encoder therein as the backbone network of a downstream recognition model to transfer the rich power semantic knowledge learned in the pre-training phase, specifically comprising: First, we use the three pre-training tasks mentioned above to pre-train CN-CLIP specifically for the power sector, obtaining a visual base model for the task of transmission line defect recognition. The total loss calculation formula for the pre-training stage is: l pre-train =λ1l ITC +λ2l CTM +λ3l DNC (18) Where, l pre-train is the total loss in the pre-training stage, and λ1, λ2, and λ3 are coefficients used to adjust the proportion of the three losses in the total loss; Then, the image encoder in the visual base model for the transmission line defect recognition task is used as the backbone network of the downstream defect recognition model, connected to the task-specific head network for performing the defect recognition task, and the downstream defect recognition model is constructed.

7. The method for identifying power line defects based on multimodal contrastive learning according to claim 6, wherein the design employs a fine-tuning strategy based on contrastive learning, adds ITC to the downstream transmission line defect identification task, uses the two tasks to fine-tune the entire recognition model to alleviate overfitting problems in the fine-tuning stage, and trains the final transmission line defect identification model. Specifically, the method comprises: When the downstream transmission line defect recognition task is a defect classification task, first, the image feature matrix output by the image encoder is passed to the task-specific head network to obtain the defect classification task output. The corresponding classification loss calculation formula is: Where, is the label that the i-th sample belongs to the j-th category, is the model's predicted probability that the i-th sample belongs to the j-th category, c is the category of the defect classification task, l CLS is the classification loss; Secondly, fill the predefined template or match the predefined description with the category label of the downstream defect classification dataset, generate a corresponding text for each image, pass all the texts into the text encoder to obtain the text feature matrix, and calculate the ITC loss according to formula (2)-formula (4). The total loss in the fine-tuning stage is the sum of the two task losses, which is: l fine-tune ＝l CLS +l ITC (20) Where, l fine-tune is the total loss in the fine-tuning phase; When the downstream transmission line defect recognition task is a defect detection task, first, the instances in the downstream defect detection dataset are cropped into instance-level images of various scales according to the annotation boxes. Then, the backbone network and the classification branch of the task-specific head network of the defect detection model are fine-tuned using the method based on the case where the downstream transmission line defect recognition task is a defect classification task. The loss at this stage is calculated as: Where, is the fine-tuning loss of the first stage; Then, ITC is removed and the entire defect detection model is fine-tuned using two tasks, defect classification and bounding box regression, on the original downstream defect detection dataset. The loss at this stage is calculated as: Where, is the fine-tuning loss of the second stage, l BBOX is the bounding box regression loss; Finally, the text encoder is removed to obtain the final transmission line defect recognition model. During model inference, the image encoder and the task-specific head network are used to infer the test image to obtain the transmission line defect recognition output, which is: Output=Taskhead(f i (x)) (23) Where x is the test image input, Output is the transmission line defect recognition output, and Taskhead(·) is the task-specific head network.

8. An electronic device, characterized in that: The electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement the transmission line defect identification method based on multimodal contrast learning as described in any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to execute the transmission line defect identification method based on multimodal contrast learning according to any one of claims 1 to 7.

Images