A multi-modal knowledge guided transmission line defect detection method

Abstract

The embodiment of the application discloses a kind of multi-modal knowledge guided transmission line defect detection methods, including steps: constructing transmission line image-text data set, through image-text contrast learning pretraining, the relevant multi-modal knowledge of transmission line contained therein is introduced into visual basic model CLIP;The embedding matrix of CLIP model pre-encoded carrying the relevant multi-modal knowledge of transmission line is introduced into the decoder of Deformable DETR detector, to realize the detector training of multi-modal knowledge guidance;Residual structure is introduced into the decoder of Deformable DETR, to alleviate the problem that the multi-modal knowledge introduced is gradually forgotten in the attention calculation of decoder multi-layer;Design a kind of pseudo-label distillation objective function, with multi-modal knowledge constraint optimization direction in the detector training process, realize more stable training effect.The application uses multi-modal knowledge to guide the training process of transmission line defect detector, effectively improves the precision of defect detection, with good robustness and stability.

Description

Technical Field

[0001] This invention relates to the field of image analysis technology, and in particular to a multimodal knowledge-guided method for detecting defects in power transmission lines. Background Technology

[0002] Transmission lines, as vital lifelines in the power system, are crucial for maintaining power system security and ensuring power quality. Various components within transmission lines form the foundation for their normal operation. Due to prolonged exposure to the outdoor environment and complex external factors, these components are prone to various degrees of failure, posing a threat to the stable operation of the transmission lines. However, transmission lines are often erected over long distances in complex environments, making traditional manual inspection methods inefficient and costly. In recent years, with the widespread adoption of drone-based transmission line inspection technology, using deep learning-based computer vision algorithms to identify defects in drone-collected inspection images has become the mainstream inspection method, effectively reducing the workload and maintenance costs of inspections.

[0003] However, due to the unique industry background and shooting conditions, training a defect detector for power transmission line inspection images still faces two challenges:

[0004] 1. Due to the influence of shooting environment, imaging distance and angle, the defective parts in the inspection images obtained by UAV aerial photography are often small in size, partially occluded and have complex backgrounds. This leads to insufficient visual information in the inspection images, making it impossible for the model to capture enough semantic information and power-related knowledge.

[0005] 2. There is a significant imbalance between the number of defective and normal components. In a batch of data, the number of samples for some defect categories is often less than 200. The scarcity of samples makes the model prone to overfitting to defect categories with insufficient samples, which is not conducive to the model's robust identification of complex defect situations in transmission lines.

[0006] With the rapid advancements in pre-training technology and its significant breakthroughs in computer vision and natural language processing in recent years, researchers have begun to focus on vision-language pre-training to train more general visual foundation models, such as CLIP, using massive image-text pairs of data (hundreds of millions of images) available in the internet. These vision-language pre-trained visual foundation models learn rich general semantic knowledge from aligned image-text data during pre-training, achieving good recognition results across a wide range of downstream tasks and under conditions of few or even zero samples. This provides a new approach to simultaneously addressing both of these challenges. However, directly applying vision-language pre-trained models like CLIP to specific scenarios such as power transmission line inspection remains challenging, primarily for three reasons: First, these models are pre-trained on web-crawled data, which contains only a small amount or even no power transmission line inspection data, thus failing to capture the unique diversity and complexity of such data. Second, defect data in power transmission line inspection is often scarce and relies on costly expert annotation, making it impossible to provide hundreds of millions of image-text pairs to support large-scale vision-language pre-training. Finally, the computational cost of training such models from scratch is unaffordable for most research institutions.

[0007] Therefore, under the above background, applying the large-scale visual basic model CLIP containing rich semantic knowledge to power transmission line defect detection overcomes the problem that the original CLIP is difficult to apply directly to power transmission line inspection scenarios and effectively integrates its knowledge into the power transmission line defect detector. This is of great significance for achieving higher precision defect detection results and intelligent power transmission line defect detection. Summary of the Invention

[0008] The purpose of this invention is to provide a multimodal knowledge-guided method for detecting transmission line defects, addressing issues such as the lack of transmission line-related knowledge in vision-language pre-trained visual models, the difficulty for transmission line defect detectors to effectively capture sufficient visual information from inspection images, and insufficient sample numbers for some defect categories. This invention designs a multimodal knowledge transfer method to introduce transmission line-related knowledge into the vision-language pre-trained visual model, and designs a knowledge enhancement structure based on residual connections and a multimodal knowledge-guided pseudo-label distillation objective function to further improve the accuracy of defect detection.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] A multimodal knowledge-guided method for detecting defects in transmission lines includes the following steps:

[0011] S1, annotate and crop the original transmission line inspection images to construct a transmission line image-text dataset, covering both visual and linguistic modalities, containing image-text pairs of various components and defect instances in the transmission line, and implying multimodal knowledge related to the transmission line.

[0012] S2 uses an image-text contrastive learning task to perform domain-specific pre-training on CLIP, a large-scale visual foundation model containing rich general semantic knowledge, on a transmission line image-text dataset, thereby introducing multimodal knowledge related to transmission lines into the CLIP model.

[0013] S3 uses a pre-trained CLIP model to pre-encode defect instance images and corresponding instance texts of different categories to obtain a pre-encoded embedding matrix carrying multimodal knowledge related to transmission lines.

[0014] S4 transforms the pre-encoded embedding matrix into object queries in the decoder input of the DeformableDETR detector through linear layers and copy, truncation, and splicing operations, thus introducing multimodal knowledge into the detector.

[0015] S5 supplements the output of each decoder layer with multimodal knowledge by establishing a residual connection between the output of each decoder layer and the object query transformed by the pre-encoded embedding matrix within the decoder of Deformable DETR, thus alleviating the problem of multimodal knowledge being forgotten in multi-layer attention computation.

[0016] S6. A pseudo-label distillation objective function is designed based on the pre-encoding embedding matrix. The intermediate output generated during the calculation of the pre-encoding embedding matrix is ​​fully utilized to obtain pseudo-labels based on multimodal knowledge. This realizes the explicit constraint of multimodal knowledge on the optimization direction during detector training to stabilize the training effect. The pseudo-label distillation objective function is combined with the original objective function of Deformable DETR to train the final transmission line defect detector.

[0017] An electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, the program being used to implement the above-mentioned method for identifying transmission line defects based on multimodal contrastive learning.

[0018] A storage medium, characterized in that the storage medium stores computer-executable instructions, which are used to execute the above-described method for identifying transmission line defects based on multimodal contrastive learning.

[0019] This invention discloses the following technical effects: This invention provides a multimodal knowledge-guided method for detecting transmission line defects, comprising the following steps: constructing a transmission line image-text dataset, containing various image-text pairs related to transmission line components and defects, for pre-training a visual foundation model containing rich multimodal knowledge related to transmission lines; using a large-scale visual foundation model CLIP for image-text contrast learning pre-training on the transmission line image-text dataset, introducing multimodal knowledge into the CLIP model; using the pre-trained CLIP model to pre-encode different categories of defect instance images and corresponding instance texts cropped from the target transmission line defect detection dataset, obtaining a pre-encoded embedding matrix carrying multimodal knowledge related to transmission lines, for inputting the knowledge into a defect detector based on Deformable DETR; and transforming the pre-encoded embedding matrix through linear layers and copying, cropping, and splicing operations to convert it into an input Deformable DETR model. The DETR decoder uses object queries to input multimodal knowledge into the detector. A decoder structure based on residual connections is designed to enhance multimodal knowledge at each decoder layer, mitigating the problem of multimodal knowledge being gradually forgotten after attention calculations across multiple decoder layers. A pseudo-label distillation objective function based on a pre-encoding embedding matrix is ​​designed to explicitly guide and constrain the detector training process with multimodal knowledge, resulting in more stable training performance. This invention applies the CLIP algorithm to transmission line defect detection, effectively improving the accuracy of transmission line defect detection by introducing transmission line-related multimodal knowledge contained in image-text pairs. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a multimodal knowledge-guided method for detecting defects in transmission lines according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the overall training process according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram illustrating the calculation and transformation method of the precoding embedding matrix according to an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of the pseudo-label distillation target in an embodiment of the present invention;

[0025] Figure 5 This is a diagram illustrating the effect of defect detection in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The purpose of this invention is to provide a multimodal knowledge-guided method for detecting transmission line defects, which solves the problems of lack of transmission line-related knowledge in CLIP, difficulty in effectively capturing sufficient visual information from inspection images, and insufficient sample numbers for some defect categories, thereby improving the accuracy of transmission line defect detection.

[0028] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] like Figure 1 As shown, the flowchart of a multimodal knowledge-guided transmission line defect detection method provided by the present invention includes the following steps:

[0030] S1, annotate and crop the original transmission line inspection images to construct a transmission line image-text dataset, covering both visual and linguistic modalities, containing image-text pairs of various components and defect instances in the transmission line, and implying multimodal knowledge related to the transmission line.

[0031] S2 uses an image-text contrastive learning task to perform domain-specific pre-training on CLIP, a large-scale visual foundation model containing rich general semantic knowledge, on a transmission line image-text dataset, thereby introducing multimodal knowledge related to transmission lines into the CLIP model.

[0032] S3, using a pre-trained CLIP model to pre-encode defect instance images and corresponding instance texts of different categories, to obtain a pre-encoded embedding matrix carrying multimodal knowledge related to transmission lines;

[0033] S4 transforms the pre-encoded embedding matrix into object queries in the decoder input of the DeformableDETR detector through linear layers and copy, truncation, and splicing operations, thus introducing multimodal knowledge into the detector.

[0034] S5 supplements the output of each decoder layer with multimodal knowledge by establishing a residual connection between the output of each decoder layer and the object query transformed by the pre-encoded embedding matrix within the decoder of Deformable DETR, thus alleviating the problem of multimodal knowledge being forgotten in multi-layer attention computation.

[0035] S6. A pseudo-label distillation objective function is designed based on the pre-encoding embedding matrix. The intermediate output generated during the calculation of the pre-encoding embedding matrix is ​​fully utilized to obtain pseudo-labels based on multimodal knowledge. This realizes the explicit constraint of multimodal knowledge on the optimization direction during detector training to stabilize the training effect. The pseudo-label distillation objective function is combined with the original objective function of Deformable DETR to train the final transmission line defect detector.

[0036] Traditional supervised pre-trained visual foundation models typically use a limited amount of general-domain image-label pairs for pre-training on image classification tasks. This results in pre-trained models with limited generalization ability and a lack of knowledge related to power transmission lines. In power transmission line defect detection tasks where visual information is limited and defect samples are insufficient, they can only achieve suboptimal results. In contrast, existing vision-language pre-trained visual foundation models are generally pre-trained on hundreds of millions of general-domain image-text pairs, learning rich general semantic knowledge and exhibiting strong generalization ability in downstream tasks. While offering greater advantages over supervised pre-trained visual foundation models, they also lack knowledge related to power transmission lines. Therefore, to further incorporate multimodal knowledge related to power transmission lines into vision-language pre-trained visual foundation models, it is necessary to construct a large-scale dataset containing multiple categories of image-text pairs of power transmission line components and defects for pre-training on downstream tasks related to power transmission lines. Therefore, in step S1, the original transmission line inspection images are labeled and cropped to construct a transmission line image-text dataset, covering both visual and linguistic modalities. This dataset contains image-text pairs of various components and defect instances within the transmission line, and embodies multimodal knowledge related to the transmission line, specifically including:

[0037] First, label and crop the original transmission line inspection images to obtain example images of different types of transmission line components and defects with category labels such as "normal vibration damper" and "vibration damper interlacing".

[0038] Then, the predefined general description template is filled with the category labels corresponding to the instance images, or a predefined category-specific description is matched to generate a matching instance text for each instance image, thus completing the construction of the transmission line image-text dataset. It is important to note that in the data used in traditional supervised pre-training, each sample consists of an image-label pair, involving only a single visual modality; while in the transmission line image-text dataset constructed by this method, each sample consists of an instance image and its corresponding instance text, including both visual and linguistic modalities. This can provide the model with rich multimodal knowledge related to transmission lines. For example, by combining an instance image about "interleaved vibration dampers" and its corresponding instance text "two vibration dampers on the same conductor are close together," the model can learn the multimodal knowledge required to identify the "interleaved vibration damper" defect while aligning the two modalities. That is, when the three conditions of "two vibration dampers," "on the same conductor," and "close together" are met simultaneously, the "interleaved vibration damper" defect can be identified.

[0039] In this approach, considering the prohibitive cost of training a CLIP model from scratch, which is unaffordable for most research institutions, and the potential for damaging the original pre-trained representations when modifying the pre-training task by introducing new transmission line-related multimodal knowledge onto existing open-source CLIP pre-trained weights, specifically training a randomly initialized CLIP model from scratch requires hundreds of millions of image-text pairs and substantial computational resources, which are currently unavailable. Furthermore, the open-source CLIP pre-trained weights are obtained through an image-text contrast learning task on a general domain of 400 million image-text pairs. Introducing transmission line-related multimodal knowledge onto these weights using image-text matching, masked language modeling, or other common visual-language pre-training tasks would actually destroy the original pre-trained representations. Therefore, using open-source pre-trained weights to initialize the CLIP model and then pre-training it on a transmission line-related image-text dataset using an image-text contrast learning task is a more efficient and feasible method. This allows the CLIP model to learn transmission line-related multimodal knowledge on top of existing general semantic knowledge while avoiding the destruction of the original pre-trained representations. Therefore, in step S2, an image-text contrast learning task is used to perform domain-specific pre-training on the large-scale visual foundation model CLIP, which contains rich general semantic knowledge, on the transmission line image-text dataset. This introduces multimodal knowledge related to transmission lines into the CLIP model, specifically including:

[0040] First, a batch of image-text pairs sampled from the transmission line image-text dataset is input into the CLIP model. The visual encoder and speech encoder in the CLIP model are used to embed features into the images and text respectively, outputting the image embedding matrix and the corresponding text embedding matrix, as shown in the formula:

[0041] V = e v (I); L = e l (T) (1)

[0042] In the formula, I and T represent a batch of images and corresponding text input to the CLIP model, respectively, and e v (·) and e l (·) represent the visual encoder and language encoder in the CLIP model, respectively, and V and L represent the image embedding matrix and text embedding matrix output by the CLIP model, respectively.

[0043] Then, based on the cosine similarity between the image embedding matrix and the corresponding text embedding matrix, the image-text contrastive learning loss is calculated, as follows:

[0044]

[0045] L pretrain =(L I→T +L T→I ) / twenty four)

[0046] In the formula, v i and l i Let v represent the i-th embedding vector in V and L respectively. j and l j Let V and L represent the j-th embedding vectors respectively, b be the batch size, τ be the temperature coefficient, and L be the value of the embedding vector. I→T and L T→I L represents the image-text contrast learning loss for image-to-text and text-to-image learning, respectively. pretrain This represents the total loss during the pre-training phase. The process described above for pre-training the CLIP model on the transmission line image-text dataset using an image-text contrast learning task can be described as follows: Figure 2 The left side indicates.

[0047] In this method, considering that the image-text contrast learning pre-training process uses instance-level image-text pairs that only describe a single component or defect instance, the pre-trained CLIP model lacks fine-grained perceptual ability and is not suitable as the backbone network of the detector for directly embedding the features of inspection images containing numerous instances in the target transmission line defect detection dataset. However, the pre-trained CLIP model can better embed global features of images and text describing a single instance. Therefore, this model can be used to perform feature embedding on the defect instance images and corresponding instance texts cropped from the target transmission line defect detection dataset to obtain a pre-encoded embedding matrix, which helps the detector better query related instances in the image. Therefore, in step S3, the pre-trained CLIP model is used to pre-encode defect instance images and corresponding instance texts of different categories to obtain a pre-encoded embedding matrix carrying multimodal knowledge related to the transmission line, specifically including:

[0048] First, defect instance images from the target transmission line defect detection dataset are cropped according to category to obtain a defect instance image set. Simultaneously, a collection of defect instance texts is obtained by populating a predefined general description template with the category name or matching a predefined category-specific description. Where k is the number of categories in the target transmission line defect detection dataset. and Let represent the subset of defect instance images and the subset of defect instance texts corresponding to the i-th category, respectively.

[0049] Secondly, the visual encoder and language encoder of the pre-trained CLIP model were used to process D. I and D T For each subset of defect categories, feature embedding is performed to obtain the corresponding pre-encoded embedding vector set, as shown in the formula:

[0050]

[0051] In the formula, and These are the visual encoder and language encoder in the pre-trained CLIP model, respectively. and These are the pre-encoded image embedding vector set and the pre-encoded text embedding vector set, respectively. and Let represent the precoded image embedding vector subset and the precoded text embedding vector subset corresponding to the i-th category, respectively.

[0052] Next, for E respectively I and E TRandom sampling is performed, and a vector is selected from the subset of embedding vectors corresponding to each defect category to obtain the precoded image embedding matrix and the precoded text embedding matrix, as shown in the formula:

[0053]

[0054] In the formula, Sample(·) represents the sampling function. and These are the precoded image embedding matrix and the precoded text embedding matrix, respectively. and They represent from and The precoded image embedding vector and precoded text embedding vector corresponding to the i-th category obtained from sampling.

[0055] Finally, the precoded image embedding matrix and the precoded text embedding matrix are fused by mask summation to obtain the precoded embedding matrix carrying multimodal knowledge related to transmission lines, as shown in the formula:

[0056]

[0057] In the formula, M is a mask matrix randomly composed of 0 and 1 vectors. The pre-encoding embedding matrix is ​​used to carry multimodal knowledge related to transmission lines, where d CLIP Output the dimension of the embedding vector for the CLIP model. The precoding process described above is as follows: Figure 3 As shown below, after obtaining the pre-encoding embedding matrices for the two modalities using the image and text encoders in the pre-trained CLIP model, the pre-encoding embedding matrix carrying the multimodal knowledge related to the transmission line can be obtained by mask summation.

[0058] In this method, considering that the shape of the pre-encoded embedding matrix carrying multimodal knowledge related to transmission lines obtained directly from the pre-trained CLIP model does not conform to the shape required by the decoder input object query in the Deformable DETR detector, step S4 converts the pre-encoded embedding matrix into the object query of the decoder input part in the Deformable DETR detector through linear layers and copying, truncation, and splicing operations, thus transmitting the multimodal knowledge into the detector. Specifically, this includes:

[0059] First, treat E as a group of row vectors. Use a learnable mapping layer to map the dimensions of each row vector in E to the object query dimension d that the Deformable DETR can accept, thus obtaining an initial object query with multimodal knowledge. The formula is as follows:

[0060]

[0061] In the formula, Linear(·) represents a linear layer. For initial object queries with multimodal knowledge.

[0062] Then, through the Perform copying, truncating, and concatenation, expanding the number of row vectors to the number of object queries n specified by Deformable DETR, using the following formula:

[0063]

[0064] In the formula, Copy(·,·) represents a copy function that copies the input matrix a specified number of times, Fetch(·,·) represents a truncation function that extracts a specified number of rows from the beginning of the input matrix, Concat(·,·) represents a concatenation function, z is the quotient of n divided by k, and y is the remainder of n divided by k. Query for the copied object. For querying the extracted objects. Query for objects whose shapes conform to the Deformable DETR input requirement. The above precoding embedding matrix transformation process is as follows: Figure 3 As shown above, by transforming the row and column dimensions of the precoding embedding matrix, an object query that can be used as input to the Deformable DETR decoder is finally obtained, thus achieving the goal of passing multimodal knowledge into the detector.

[0065] In this method, considering that the Deformable DETR decoder contains multiple layers of self-attention and cross-attention structures, multimodal knowledge is only passed into the first decoder layer and lacks an effective information enhancement method. Therefore, a large amount of knowledge is easily forgotten after passing through multiple decoder layers. This method establishes a residual connection between the output of each decoder layer and the input object query, introducing multimodal knowledge into each decoder layer, which effectively avoids the above problem. Therefore, in step S5, by establishing a residual connection between the output of each decoder layer and the object query obtained by transformation from the pre-encoded embedding matrix within the Deformable DETR decoder, multimodal knowledge is supplemented to the output of each decoder layer, alleviating the problem of multimodal knowledge being forgotten in multi-layer attention calculations. Specifically, this includes:

[0066] First, Q m Adding the original object query, which is randomly initialized in the Deformable DETR, to obtain the final object query used as input to the first decoder layer of the Deformable DETR, the formula is as follows:

[0067]

[0068] In the formula, Qo Query the original objects that are randomly initialized in the Deformable DETR. The input is the object query for the first decoder layer.

[0069] Then, based on the original decoder, a residual connection is introduced to connect the object query output by each decoder layer with the Q-value. m A weighted summation is performed to calculate the multimodal knowledge-enhanced object query corresponding to this layer, which serves as the final output object query of this layer and the input of the next decoder layer. The formula is as follows:

[0070]

[0071] In the formula, FFN(·) represents a feedforward network, CA(·) represents a cross-attention module, MHSA(·) represents a self-attention module, and Q... l For querying the original object output by the l-th decoder layer, and X represents the multimodal knowledge-enhanced object query of the l-th and (l-1)-th decoder layers, respectively. X is the feature map output by the Deformable DETR encoder, and α is the learnable weight.

[0072] In this method, considering that the Deformable DETR detector, which incorporates a pre-encoding embedding matrix carrying multimodal knowledge related to transmission lines and a decoder structure based on residual connections, lacks effective constraints during training, this is detrimental to its learning of multimodal knowledge and achieving stable training results. The pre-encoding embedding matrix can provide a supervisory signal based on multimodal knowledge, enabling explicit guidance and constraints from multimodal knowledge on the detector training process. Therefore, in step S6, a pseudo-label distillation objective function is designed based on the pre-encoding embedding matrix. This fully utilizes the intermediate output generated during the calculation of the pre-encoding embedding matrix to obtain pseudo-labels based on multimodal knowledge, achieving explicit constraints from multimodal knowledge on the optimization direction during detector training to stabilize the training effect. The pseudo-label distillation objective function is then combined with the original objective function of the Deformable DETR detector to train the final transmission line defect detector, specifically including:

[0073] First, using the precoding process described above to obtain and The formula for calculating pseudo-labels based on multimodal knowledge is as follows:

[0074]

[0075] In the formula, ⊙ represents the dot product operation. These are initial pseudo-labels based on multimodal knowledge.

[0076] Secondly, Treat it as a group of row vectors, and use the extended... For Q m The method for The number of row vectors in the array is expanded to be equal to the number of object queries n specified by Deformable DETR, as shown in the formula:

[0077]

[0078] In the formula, These are the copied pseudo tags. These are the extracted pseudo-tags. This is the expanded pseudo-tag.

[0079] Next, a sparse pseudo-label assignment strategy is designed, which adds pseudo-label supervision only to the best matching object query output by the Deformable DETR header and calculates the loss, as follows:

[0080]

[0081] In the formula, L pld Distillation loss due to false labeling For Y p The j-th element of the i-th pseudo-label vector, where w is the total number of decoder layers. Let ε be the j-th element of the i-th object query vector output by the last decoder layer. ε represents the result of the indicator function calculation. ε is 1 if and only if the object query vector matches a certain benchmark true label and the benchmark true label and the pseudo-label correspond to different categories; otherwise, ε is 0. The process by which the pseudo-label distillation objective function achieves its supervisory role is as follows: Figure 4 As shown.

[0082] Finally, a pseudo-label distillation loss is added to the original target classification loss and bounding box regression loss of Deformable DETR to explicitly guide the Deformable DETR training process with multimodal knowledge, resulting in more stable training performance. The formula is as follows:

[0083] L det =λ cls L cls +λ box L box +λ pld L pld (19)

[0084] In the formula, L cls For the target classification loss, L box For bounding box regression loss, L det λ is the total loss for training the detector. cls, λ box and λ pld Used to adjust L respectively cls L box and L pld In L det The proportion of [something] in the training process. The complete detector structure corresponding to this training process is as follows: Figure 2 As shown on the right.

[0085] The detection effect of the method of the present invention is as follows: Figure 5 As shown, this invention, based on the Deformable DETR detector, introduces a CLIP precoding embedding matrix containing multimodal knowledge related to transmission lines, enabling the detector to learn richer semantic information and prior knowledge related to transmission line components and defects. To address the problem of knowledge forgetting easily caused by the original decoder structure, a knowledge enhancement structure based on residual connections is added to the decoder to alleviate the problem of forgetting multimodal knowledge input to the decoder. To address the problem of unstable training results due to the lack of explicit constraints in the original objective function, a pseudo-label distillation objective function based on the precoding embedding matrix is ​​designed, utilizing multimodal knowledge to supervise the detector's optimization process, achieving better and more stable training results. It is evident that the method described in this invention improves the detection accuracy of various defects while ensuring stable training results for the defect detector. This invention effectively improves the detection performance of the Deformable DETR detector by introducing multimodal knowledge contained in transmission line-related image-text pairs, providing a new approach for the research of training high-performance transmission line defect detectors.

[0086] In addition, this application also provides an apparatus, electronic equipment and storage medium corresponding to the aforementioned multimodal knowledge-guided transmission line defect detection method, as detailed below:

[0087] The device includes:

[0088] The S1 module is used to annotate and crop the original transmission line inspection images, constructing a transmission line image-text dataset that covers both visual and linguistic modalities. It contains image-text pairs of various components and defect instances in the transmission lines, and contains multimodal knowledge related to transmission lines.

[0089] The S2 module is used to perform domain-specific pre-training on the large-scale visual foundation model CLIP, which contains rich general semantic knowledge, on the transmission line image-text contrast learning task, thereby introducing multimodal knowledge related to transmission lines into the CLIP model.

[0090] The S3 module is used to pre-encode defect instance images and corresponding instance texts of different categories using a pre-trained CLIP model, thereby obtaining a pre-encoded embedding matrix that carries multimodal knowledge related to transmission lines.

[0091] The S4 module is used to convert the pre-encoded embedding matrix into object queries in the decoder input of the Deformable DETR detector through linear layers and copy, truncation, and splicing operations, thus passing multimodal knowledge into the detector.

[0092] The S5 module is used to supplement the output of each decoder layer with multimodal knowledge by establishing a residual connection between the output of each decoder layer and the object query transformed by the pre-encoded embedding matrix within the decoder of Deformable DETR, thereby mitigating the problem of multimodal knowledge being forgotten in multi-layer attention computation.

[0093] The S6 module is used to design a pseudo-label distillation objective function based on the pre-encoding embedding matrix. It makes full use of the intermediate output generated when calculating the pre-encoding embedding matrix to obtain pseudo-labels based on multimodal knowledge. It realizes the explicit constraint of multimodal knowledge on the optimization direction during detector training to stabilize the training effect. The pseudo-label distillation objective function is combined with the original objective function of DeformableDETR to obtain the final transmission line defect detector.

[0094] An electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, the program being used to implement the multimodal knowledge-guided transmission line defect detection method.

[0095] A storage medium storing computer-executable instructions for executing the aforementioned multimodal knowledge-guided transmission line defect detection method.

[0096] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multimodal knowledge-guided method for detecting defects in transmission lines, characterized in that, Includes the following steps: S1, annotate and crop the original transmission line inspection images to construct a transmission line image-text dataset, covering both visual and linguistic modalities, containing image-text pairs of various components and defect instances in the transmission line, and containing multimodal knowledge related to the transmission line; S2 uses an image-text contrast learning task to perform domain-specific pre-training on CLIP, a large-scale visual foundation model containing rich general semantic knowledge, on a transmission line image-text dataset, thereby introducing multimodal knowledge related to transmission lines into the CLIP model. S3, using a pre-trained CLIP model to pre-encode defect instance images and corresponding instance texts of different categories, to obtain a pre-encoded embedding matrix carrying multimodal knowledge related to transmission lines; S4 transforms the pre-encoded embedding matrix into object queries in the decoder input of the Deformable DETR detector through linear layers and copy, truncation, and splicing operations, thus introducing multimodal knowledge into the detector; S5, by establishing a residual connection between the output of each decoder layer and the object query obtained by the transformation of the pre-encoded embedding matrix within the decoder of Deformable DETR, supplements the output of each decoder layer with multimodal knowledge, thus alleviating the problem of multimodal knowledge being forgotten in multi-layer attention computation; S6. A pseudo-label distillation objective function is designed based on the pre-encoding embedding matrix. The intermediate output generated during the calculation of the pre-encoding embedding matrix is ​​fully utilized to obtain pseudo-labels based on multimodal knowledge. This realizes the explicit constraint of multimodal knowledge on the optimization direction during detector training to stabilize the training effect. The pseudo-label distillation objective function is combined with the original objective function of Deformable DETR to train the final transmission line defect detector.

2. The multimodal knowledge-guided transmission line defect detection method according to claim 1, wherein the annotation and cropping of the original transmission line inspection images, constructing a transmission line image-text dataset covering both visual and linguistic modalities, containing image-text pairs of various components and defect instances in the transmission line, and embodying multimodal knowledge related to the transmission line, specifically includes: First, the original transmission line inspection images are labeled and cropped to obtain instance images of different types of transmission line components and defects with category labels; Then, the predefined general description template is filled with the category label corresponding to the instance image or the predefined category-specific description is matched to generate the instance text that matches each instance image, thus completing the construction of the transmission line image-text dataset. Each image-text pair contains multimodal knowledge related to the component or defect category to which the instance belongs.

3. The multimodal knowledge-guided transmission line defect detection method according to claim 1, wherein the image-text contrast learning task is used to perform domain-specific pre-training on a large-scale visual foundation model CLIP containing rich general semantic knowledge on a transmission line image-text dataset, and multimodal knowledge related to transmission lines is introduced into the CLIP model, specifically including: First, a batch of image-text pairs sampled from the transmission line image-text dataset is input into the CLIP model. The visual encoder and speech encoder in the CLIP model are used to embed features into the images and text respectively, outputting the image embedding matrix and the corresponding text embedding matrix, as shown in the formula: V=e v (I);L=e l (T) (1) In the formula, I and T represent a batch of images and corresponding text input to the CLIP model, respectively, and e v (·) and e l (·) represent the visual encoder and language encoder in the CLIP model, respectively, and V and L represent the image embedding matrix and text embedding matrix output by the CLIP model, respectively. Then, based on the cosine similarity between the image embedding matrix and the corresponding text embedding matrix, the image-text contrastive learning loss is calculated, as follows: L pretrain (L I→T +L T→I ) / 2 (4) In the formula, v i and l i Let v represent the i-th embedding vector in V and L respectively. j and l j Let V and L represent the j-th embedding vectors respectively, b be the batch size, τ be the temperature coefficient, and L be the value of the embedding vector. I→T and L T→I L represents the image-text contrast learning loss for image-to-text and text-to-image learning, respectively. pretrain This represents the total loss during the pre-training phase.

4. The multimodal knowledge-guided transmission line defect detection method according to claim 1, wherein the pre-trained CLIP model is used to pre-encode defect instance images and corresponding instance texts of different categories to obtain a pre-encoded embedding matrix carrying multimodal knowledge related to the transmission line, specifically including: First, defect instance images from the target transmission line defect detection dataset are cropped according to category to obtain a defect instance image set. Simultaneously, a collection of defect instance texts is obtained by populating a predefined general description template with the category name or matching a predefined category-specific description. Where k is the number of categories in the target transmission line defect detection dataset. and Let these represent the subset of defect instance images and the subset of defect instance texts corresponding to the i-th category, respectively. Secondly, the visual encoder and language encoder of the pre-trained CLIP model were used to process D. I and D T For each subset of defect categories, feature embedding is performed to obtain the corresponding pre-encoded embedding vector set, as shown in the formula: In the formula, and These are the visual encoder and language encoder in the pre-trained CLIP model, respectively. and These are the pre-encoded image embedding vector set and the pre-encoded text embedding vector set, respectively. and Let represent the precoded image embedding vector subset and the precoded text embedding vector subset corresponding to the i-th category, respectively; Next, for E respectively I and E T Random sampling is performed, and a vector is selected from the subset of embedding vectors corresponding to each defect category to obtain the precoded image embedding matrix and the precoded text embedding matrix, as shown in the formula: In the formula, Sample(·) represents the sampling function. and These are the precoded image embedding matrix and the precoded text embedding matrix, respectively. and They represent from and The precoded image embedding vector and precoded text embedding vector corresponding to the i-th category obtained from sampling; Finally, the precoded image embedding matrix and the precoded text embedding matrix are fused by mask summation to obtain the precoded embedding matrix carrying multimodal knowledge related to transmission lines, as shown in the formula: In the formula, M is a mask matrix randomly composed of 0 and 1 vectors. The pre-encoding embedding matrix is ​​used to carry multimodal knowledge related to transmission lines, where d CLIP Output the dimension of the embedding vector for the CLIP model.

5. The multimodal knowledge-guided transmission line defect detection method according to claim 1, wherein the step of converting the precoded embedding matrix into an object query in the decoder input of a Deformable DETR detector through linear layers and copying, truncation, and splicing operations, and inputting multimodal knowledge into the detector, specifically includes: First, treat E as a group of row vectors. Use a learnable mapping layer to map the dimensions of each row vector in E to the object query dimension d that the Deformable DETR can accept, thus obtaining an initial object query with multimodal knowledge. The formula is as follows: In the formula, Linear(·) represents a linear layer. For initial object queries with multimodal knowledge; Then, through the Perform copying, truncating, and concatenation, expanding the number of row vectors to the number of object queries n specified by DeformableDETR, using the following formula: In the formula, Copy(·,·) represents a copy function that copies the input matrix a specified number of times, Fetch(·,·) represents a truncation function that extracts a specified number of rows from the beginning of the input matrix, Concat(·,·) represents a concatenation function, z is the quotient of n divided by k, and y is the remainder of n divided by k. Query for the copied object. For querying the extracted objects. Query for objects whose shape conforms to the Deformable DETR input requirement.

6. The multimodal knowledge-guided transmission line defect detection method according to claim 1, wherein establishing a residual connection between the output of each decoder layer and the object query obtained by the pre-encoding embedding matrix transformation within the decoder of Deformable DETR, supplementing the output of each decoder layer with multimodal knowledge, and alleviating the problem of multimodal knowledge being forgotten in multi-layer attention calculation, specifically includes: First, Q m Adding the original object query, which is randomly initialized in the Deformable DETR, to obtain the final object query used as input to the first decoder layer of the Deformable DETR, the formula is as follows: In the formula, Q o Query the original objects that are randomly initialized in the Deformable DETR. The input is the object query for the first decoder layer; Then, based on the original decoder, a residual connection is introduced to connect the object query output by each decoder layer with the Q-value. m A weighted summation is performed to calculate the multimodal knowledge-enhanced object query corresponding to this layer, which serves as the final output object query of this layer and the input of the next decoder layer. The formula is as follows: In the formula, FFN(·) represents a feedforward network, CA(·) represents a cross-attention module, MHSA(·) represents a self-attention module, and Q... l For querying the original object output by the l-th decoder layer, and X represents the multimodal knowledge-enhanced object query of the l-th and (l-1)-th decoder layers, respectively. X is the feature map output by the Deformable DETR encoder, and α is the learnable weight.

7. The method for detecting transmission line defects guided by multimodal knowledge according to claim 1, wherein a pseudo-label distillation objective function is designed based on a pre-encoding embedding matrix, and the intermediate output generated during the calculation of the pre-encoding embedding matrix is ​​fully utilized to obtain pseudo-labels based on multimodal knowledge, thereby realizing explicit constraints of multimodal knowledge on the optimization direction during detector training to stabilize the training effect, and the pseudo-label distillation objective function is combined with the original objective function of Deformable DETR to train the final transmission line defect detector, specifically including: First, using the precoding process described above to obtain and The formula for calculating pseudo-labels based on multimodal knowledge is as follows: In the formula, ⊙ represents the dot product operation. Initial pseudo-labels based on multimodal knowledge; Secondly, Treat it as a group of row vectors, and use the extended... For Q m The method for The number of row vectors in the array is expanded to be equal to the number of object queries n specified by Deformable DETR, as shown in the formula: In the formula, These are the copied pseudo tags. These are the extracted pseudo-tags. For expanded pseudo-tags; Furthermore, a sparse pseudo-label assignment strategy is designed, which adds pseudo-label supervision only to the best-matching object query output by the Deformable DETR header and calculates the loss, as follows: In the formula, L pld Distillation loss due to false labeling For Y p The j-th element of the i-th pseudo-label vector, where w is the total number of decoder layers. ε represents the j-th element of the i-th object query vector output by the last decoder layer. ε indicates the result of the indicator function. ε is 1 if and only if the object query vector matches a certain benchmark true label and the benchmark true label and the pseudo label correspond to different categories. Otherwise, ε is 0. Finally, a pseudo-label distillation loss is added to the original target classification loss and bounding box regression loss of Deformable DETR to explicitly guide the Deformable DETR training process with multimodal knowledge, resulting in more stable training performance. The formula is as follows: L det =λ cls L cls +λ box L box +λ pld L pld (19) In the formula, L cls For the target classification loss, L box For bounding box regression loss, L det λ is the total loss for training the detector. cls , λ box and λ pld Used to adjust L respectively cls L box and L pld In L det The proportion it accounts for.

8. An electronic device, characterized in that, The electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, the program being used to implement the multimodal knowledge-guided transmission line defect detection method according to any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores computer-executable instructions for executing a multimodal knowledge-guided transmission line defect detection method according to any one of claims 1-7.

Images