An end-to-end visual language model, detection system and detection method for a power patrol scene

Abstract

This invention provides an end-to-end visual language model, detection system, and detection method for power grid inspection scenarios. The visual language model includes an image branch and a text branch. The image branch extracts initial features from the image using a ResNet50 backbone network, then fuses multi-scale features through an improved feature pyramid, generates candidate regions using a region proposal network, and finally completes target detection and localization through region of interest pooling, feature refinement modules, and classification and regression heads. The text branch first embeds the image category into different positions of the text prompt to construct learnable text prompts, then obtains text features through a text encoder, and finally calculates the similarity between the text features and image region features for region classification. This invention's visual language model effectively improves the accuracy of defect detection in power transmission line electrical components, providing reliable technical support for intelligent power grid inspection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more particularly to an end-to-end visual language model and detection method for power inspection scenarios. This method can detect defects in power devices in transmission lines, thereby helping power grid workers to inspect the lines in real time. Background Technology

[0002] With the rapid development of China's power industry and the continuous expansion of its power grid, transmission lines, as a crucial component of the grid, play a vital role in grid security. Transmission lines are constantly exposed to the natural environment and are inevitably affected by geographical conditions and climate, such as lightning strikes, snowstorms, and hail, which accelerate the aging and damage of electrical components. Furthermore, because transmission lines are erected in the air with few fixed support points and are heavy, they are subjected to both internal stress and external mechanical tension over a long period, leading to friction damage, structural fractures, and component detachment. If these problems are not promptly inspected and addressed, they can cause serious damage to devices and equipment in the power system. Therefore, regular inspections of electrical components, identifying and replacing faulty parts, are essential for ensuring grid security.

[0003] Computer vision has undergone several stages of development and fluctuations. Various deep learning-based object detection algorithms have not only been widely applied in fields such as medicine and transportation, but have also made power grid inspection more efficient and intelligent. Deep learning-based computer vision technology is widely used in power device defect detection during transmission line inspections, significantly improving the efficiency and accuracy of defect detection and providing crucial support for the stable operation of transmission lines. However, current power device defect detection methods still face many challenges: First, power devices in transmission lines exhibit significant scale differences, making feature fusion using traditional feature pyramid networks prone to feature conflicts and loss of detail; second, transmission lines contain a large amount of complex background interference, making it difficult to fully extract features from smaller defects; finally, existing visual language models typically use fixed text prompt templates, limiting the model's generalization ability in open scenes. Summary of the Invention

[0004] The technical solution of this invention is used to solve the problem that existing methods are not effective in detecting defects of different scales in complex backgrounds.

[0005] The present invention solves the above-mentioned technical problems through the following technical means:

[0006] An end-to-end visual language model for power inspection scenarios includes an image branch and a text branch. The image branch extracts initial features from the image using a ResNet50 backbone network, then fuses multi-scale features through an improved feature pyramid, generates candidate regions using a region proposal network, and finally completes target detection and localization through region pooling, feature refinement modules, and classification and regression heads. The text branch first embeds the image category into different positions of the text prompt to construct a learnable text prompt, then obtains text features through a text encoder, and finally calculates the similarity between the text features and image region features for region classification.

[0007] Furthermore, the improved feature pyramid includes a fusion module, which performs nearest-neighbor interpolation upsampling on high-level features and 1×1 convolution operation on low-level features. The upsampled high-level features are then connected to the 1×1 convolutional low-level features to achieve multi-scale feature fusion. Subsequently, the fused features are further improved in spatial resolution through sub-pixel convolution to generate a higher resolution feature map. Finally, a 3×3 convolutional layer is used to refine the generated feature map.

[0008] Furthermore, the specific operation process of the feature refinement module is as follows: the input features are enlarged by depthwise convolution, depthwise dilated convolution and point convolution to extract more local detail information; then, the obtained intermediate features are fed into the coordinate attention module and the multi-head self-attention module through a fully connected layer to obtain the enhanced feature fusion output; finally, the features are normalized by the LN layer and fed into the subsequent classification head and regression head to complete the detection and localization of the target.

[0009] Furthermore, the text branch includes a text prompt module and a text encoder; in the text prompt module, text prompt vectors are dynamically generated according to the image category, and contextual vocabulary is automatically modeled through learnable continuous vectors. The learnable text vector prompts are composed of randomly initialized vectors and state text; when using a Transformer as the text encoder, a prompt vector with the same dimension as the text embedding needs to be generated. With the original embedding The elements are added bit by bit, then position encoding information is added, and finally the processed data is input into the Transformer encoder layer for calculation.

[0010] This invention also provides a power device defect detection system for power transmission lines, including the aforementioned end-to-end visual language model for power inspection scenarios. The visual language model includes an image branch and a text branch. The image branch extracts initial features from the image through a ResNet50 backbone network, then fuses multi-scale features through an improved feature pyramid, generates candidate regions using a region proposal network, and finally completes target detection and localization through region pooling, feature refinement modules, and classification and regression heads. The text branch first embeds the image category into different positions of the text prompt to construct a learnable text prompt, then obtains text features through a text encoder, and finally calculates the similarity between the text features and image region features for region classification.

[0011] Furthermore, the improved feature pyramid includes a fusion module, which performs nearest-neighbor interpolation upsampling on high-level features and 1×1 convolution operation on low-level features. The upsampled high-level features are then connected to the 1×1 convolutional low-level features to achieve multi-scale feature fusion. Subsequently, the fused features are further improved in spatial resolution through sub-pixel convolution to generate a higher resolution feature map. Finally, a 3×3 convolutional layer is used to refine the generated feature map.

[0012] Furthermore, the specific operation process of the feature refinement module is as follows: the input features are enlarged by depthwise convolution, depthwise dilated convolution and point convolution to extract more local detail information; then, the obtained intermediate features are fed into the coordinate attention module and the multi-head self-attention module through a fully connected layer to obtain the enhanced feature fusion output; finally, the features are normalized by the LN layer and fed into the subsequent classification head and regression head to complete the detection and localization of the target.

[0013] Furthermore, the text branch includes a text prompt module and a text encoder; in the text prompt module, text prompt vectors are dynamically generated according to the image category, and contextual vocabulary is automatically modeled through learnable continuous vectors. The learnable text vector prompts are composed of randomly initialized vectors and state text; when using a Transformer as the text encoder, a prompt vector with the same dimension as the text embedding needs to be generated. With the original embedding The elements are added bit by bit, then position encoding information is added, and finally the processed data is input into the Transformer encoder layer for calculation.

[0014] This invention also provides a method for detecting defects in insulator strings, comprising: acquiring images of insulator strings in transmission lines as input to a visual language model; embedding three types of status prompts—"normal insulator," "damaged skirt," and "corroded steel cap"—into a text branch, and setting corresponding learnable text prompt vectors for each; extracting multi-scale features from the insulator string images through the image branch and generating candidate regions; simultaneously, dynamically generating learnable text prompts according to preset categories and encoding them as text features through the text branch; and finally calculating the similarity between the region features and the text features to complete the classification and localization of insulator defects.

[0015] The present invention also provides a processing device, including at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the above-described method by calling the program instructions.

[0016] The advantages of this invention are: This invention proposes an end-to-end visual language model learning method for power grid inspection scenarios. By improving the feature pyramid network, it achieves adaptive fusion of cross-scale features. It constructs a feature refinement module that integrates coordinate attention mechanism and multi-head self-attention mechanism to enhance the feature representation ability of small-sized defects in complex backgrounds. Furthermore, it adopts a dynamic and learnable text prompt mechanism to achieve accurate alignment between visual and language modalities. This effectively improves the accuracy of defect detection of power devices in transmission lines and provides reliable technical support for intelligent power grid inspection. Attached Figure Description

[0017] Figure 1 This is the network architecture of the visual language model in this embodiment of the invention; Figure 2 for Figure 1 The improved Feature Pyramid Network (FM-FPN) network architecture; Figure 3 for Figure 2 The network architecture of the FM (Fast-Fast) module; Figure 4 for Figure 1 Network architecture of the mid-feature refinement module; Figure 5 for Figure 4 Network architecture of the mid-coordinate attention module; Figure 6 for Figure 4 Network architecture of multi-head self-attention modules; Figure 7 for Figure 1 Network architecture of Chinese text embedding generative network; Figure 8 for Figure 7 Network architecture of Chinese text encoder. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 This invention proposes an end-to-end visual language model for power line inspection scenarios, which mainly consists of two parts: an image branch and a text branch. Its overall architecture is as follows: Figure 1 As shown. Image branch: First, the image is processed through the backbone network ResNet50 to extract initial features. Then, multi-scale features are fused using an improved feature pyramid (FM-FPN), and candidate regions are generated using a region proposal network (RPN). Finally, target detection and localization are completed through region of interest pooling (RoI Pooling), feature refinement module (FRM), and classification and regression heads. Text branch: First, the image category is embedded into different positions of the text prompt to construct a learnable text prompt. Then, text features are obtained through a text encoder. Finally, the similarity between the text features and image region features is calculated for region classification. It is worth noting that image region features are used for bounding box regression, region classification, and similarity calculation with text features. Specifically, the region features are first input into the regression head for bounding box coordinate prediction, and L1 loss function is used for supervision. Then, the region features are processed by the classification head to generate classification scores, and the similarity between them and text features is calculated as the probability value of the corresponding category. Finally, the classification score and the similarity score are added to obtain the final image classification score, and cross-entropy loss function is used for supervision.

[0020] In traditional feature pyramid networks, high-level features typically contain more abstract semantic information, while low-level features retain richer spatial details. However, feature conflicts and loss of detail often occur during the fusion of features from different levels, affecting the effective representation of multi-scale targets. Addressing the challenge of detecting large size variations in power device defects in transmission lines, this embodiment proposes a Feature Pyramid Network (FM-FPN) structure. This structure adaptively fuses multi-scale features from different levels, effectively resolving feature conflicts and information loss, and improving the detection capability for targets at different scales.

[0021] The structure of FM-FPN is as follows Figure 2As shown, it employs both top-down and bottom-up connection methods to fully explore the contextual relationships between global semantics and local details, further improving the model's ability to identify defects at different scales. In the lateral connection part of the top-down path, each set of corresponding low-level and high-level features is input into the fusion module for processing, ultimately generating a multi-scale feature representation that can be shared by the RPN and RoI Pooling layers, enhancing the robustness of the detection model to scale changes.

[0022] To efficiently generate high-resolution feature maps, the fusion module (FM) integrates high-level and low-level features to extract semantic information at different granularities and combines sub-pixel convolution to achieve upsampling. The structure of this module is as follows: Figure 3 As shown, the specific process is as follows: First, the high-level features (outputs from deeper layers of the network) are upsampled using nearest-neighbor interpolation to improve their spatial resolution. Simultaneously, the low-level features (outputs from shallower layers of the network) are subjected to a 1×1 convolution operation to adjust the number of channels and enhance their representational power. Then, the upsampled high-level features are concatenated with the 1×1 convolutional low-level features to achieve multi-scale feature fusion. Subsequently, the fused features are further enhanced in spatial resolution through sub-pixel convolution to generate a higher-resolution feature map. Finally, a 3×3 convolutional layer is used to refine the generated feature map. The entire calculation process can be represented as:

[0023] in, This indicates the output of the fusion module. and These represent high-level features and low-level features, respectively. Indicates an upsampling operation. This represents a 1×1 convolution operation. This indicates a splicing operation. This represents a 3×3 convolution operation. This represents subpixel convolution.

[0024] Furthermore, due to the presence of a large amount of complex background interference in the defect samples of power transmission line components, the feature representation of some smaller defect regions is insufficient. To address this issue, this embodiment proposes a feature refinement module constructed from a coordinate attention mechanism and a multi-head self-attention mechanism. This module operates within the detection head after the feature map has undergone ROI Pooling, and its structure is as follows: Figure 4As shown, firstly, the input features are enlarged by depthwise convolution (DW-Conv), depthwise dilated convolution (DW-D-Conv), and pointwise convolution (1×1Conv) to increase the receptive field, thereby extracting more local detail information. Then, the obtained intermediate features are passed through a fully connected layer (FC) and fed into a coordinate attention module (CAM) and a multi-head self-attention module (MSAM) to obtain the enhanced feature fusion output. Finally, the output is normalized by an LN layer and fed into subsequent classification and regression heads to complete target detection and localization. This process can be represented as:

[0025]

[0026] in, , and These represent the input features, intermediate features, and output features, respectively. This indicates element-wise multiplication. This indicates a splicing operation.

[0027] a) Coordinate Attention Module (CAM) Coordinate attention is an attention mechanism used in deep learning that aims to incorporate positional information into channel attention to enhance the model's representational capabilities. First, the input features are pooled along the X and Y directions to generate feature maps containing horizontal and vertical coordinate information of the input feature space. Then, Conv1d is applied to locally interact with the two feature vectors, decoupling the horizontal and vertical feature vector processing to avoid the adverse effects of reduced channel dimension. Subsequently, the resulting feature vectors are processed through GN and Sigmoid activation functions to generate positional attention vectors in both directions. Finally, the horizontal and vertical positional attention vectors are multiplied to obtain the final positional attention representation.

[0028] like Figure 5 As shown, suppose there is a feature input as... Where C, H, and W represent the channel dimension, height, and width of the feature, respectively. Average pooling is performed on each channel in both the horizontal and vertical directions of the spatial dimension, and the pooling output is shown in the following formula:

[0029]

[0030] in, This represents the output at height h of the c-th channel in the horizontal direction of the feature space. This represents the output at the width w of the c-th channel in the vertical direction of the feature space.

[0031] Then, one-dimensional convolution and grouping normalization operations are used to enhance and process the positional information in the horizontal and vertical directions, thereby obtaining the positional attention representations in the horizontal and vertical directions, as shown in the following equation:

[0032]

[0033] in, , These represent positional attention representations in the horizontal and vertical directions, respectively. This represents the activation function. GN This indicates a grouping normalization operation. , These represent one-dimensional convolutions of feature vectors in the horizontal and vertical directions, respectively.

[0034] Finally, the output Y of the coordinate attention module is obtained, and its calculation formula is as follows:

[0035] b) Multi-head Self-Attention Module (MSAM) The Transformer performs exceptionally well in computer vision, primarily due to its multi-head attention mechanism. This invention introduces an additional non-linear processing layer into the original structure, thereby enhancing the model's ability to capture contextual information. This improvement not only establishes more comprehensive global associations but also effectively expands the receptive range of image features. The workflow of the multi-head self-attention module is as follows: Figure 6 As shown: First, input features The system generates three components—query (Q), key (K), and value (V)—using fully connected layers and performs linear transformations on them. Next, the transformed Q and K are multiplied by matrix multiplication and then processed through two cascaded fully connected layers, each followed by a ReLU activation function for non-linear transformation. The processed result is then multiplied by V by matrix multiplication and passed through another fully connected layer. Finally, this output is added to the original input features via a residual concatenation to obtain the final output. The entire calculation process can be represented as:

[0036] Furthermore, the overall architecture of the text embedding generative network is as follows: Figure 7As shown, it mainly consists of two parts: a text prompting module and a text encoder. In the text prompting module, the system dynamically generates text prompts based on image categories, automatically modeling contextual vocabulary through learnable continuous vectors, thus replacing the traditional manual setting method. These learnable text prompts are composed of randomly initialized vectors and state text, with different prompt content designed for normal and defective samples. Specifically, the text prompts for the text encoder can be designed in the following form:

[0037]

[0038] in, and Let M be a learnable vector, and M represent the number of learnable vectors.

[0039] like Figure 8 As shown, when using Transformer as a text encoder, a cue vector with the same dimension as the text embedding is required. With the original embedding The elements are added bit by bit, then position encoding information is added, and finally the processed data is input into the Transformer encoder layer for calculation.

[0040] In summary, this embodiment addresses the challenge of detecting defects in power devices with varying scales during power transmission line inspections by designing a novel feature pyramid network. This network employs a combination of top-down and bottom-up connection methods, integrating high-level semantic information and low-level spatial details through a fusion module. Furthermore, sub-pixel convolution is introduced to enhance feature map resolution, effectively resolving the feature conflict and detail loss issues inherent in traditional feature pyramid networks. This significantly improves the detection accuracy of defects in power devices during power transmission lines.

[0041] To address the issue of blurred features of small-sized defects caused by complex background interference during transmission line inspections, a novel feature refinement module was designed. First, the receptive field of the small defects is expanded through deep convolution. Then, a coordinate attention module is combined to extract channel features with location awareness. At the same time, a multi-head self-attention module is used to model global context dependencies, effectively improving the discriminative feature representation ability of small-sized defects in complex power grid environments.

[0042] To address the diverse nature of defects in power devices during transmission line inspections, a novel dynamic learnable text prompt generation method is designed. First, learnable text prompt vectors are generated based on the power device category and then combined with device status description text to input a text encoder. Finally, the vectors are added bit by bit to the original text embedding within a Transformer structure to achieve fine-grained semantic modeling of power devices. This effectively improves the accuracy of the model in detecting defects in various power devices under complex transmission line scenarios.

[0043] Example 2 This embodiment is based on the application of the end-to-end visual language model described in Embodiment 1 in a real-world scenario, which includes, but is not limited to, power inspection scenarios. Taking a power inspection scenario as an example, the process of detecting defects in power devices using the visual language model described in Embodiment 1 is introduced: Power line inspection scenarios include power devices such as insulator strings, grading rings, and connecting hardware, with damaged insulator skirts and corroded steel caps being key concerns. Three status indicators—"normal insulator," "damaged insulator skirts," and "corroded steel caps"—are embedded in the text branch, with corresponding learnable text indicator vectors assigned to each. During defect detection, high-resolution visible light images of the insulator strings are first collected by a drone as input to the visual language model. Then, multi-scale features are extracted through the image branch to generate candidate regions. Simultaneously, the text branch dynamically generates learnable text indicators based on preset categories and encodes them as text features. Finally, the similarity between the region features and the text features is calculated to classify and locate insulator defects.

[0044] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end visual language model for power line inspection scenarios, characterized in that, The system includes an image branch and a text branch. The image branch extracts initial features from the image using a ResNet50 backbone network, then fuses multi-scale features through an improved feature pyramid, generates candidate regions using a region proposal network, and finally completes target detection and localization through region of interest pooling, feature refinement modules, and classification and regression heads. The text branch first embeds the image category into different positions of the text prompt to construct a learnable text prompt, then obtains text features through a text encoder, and finally calculates the similarity between the text features and image region features for region classification.

2. The end-to-end visual language model for power inspection scenarios according to claim 1, characterized in that, The improved feature pyramid includes a fusion module, which performs nearest neighbor interpolation upsampling on high-level features, performs 1×1 convolution operation on low-level features, and connects the upsampled high-level features with the 1×1 convolutioned low-level features to achieve the fusion of multi-scale features. Subsequently, the fused features are further enhanced in spatial resolution through subpixel convolution to generate a higher resolution feature map; finally, a 3×3 convolutional layer is used to refine the generated feature map.

3. The end-to-end visual language model for power inspection scenarios according to claim 1 or 2, characterized in that, The specific operation process of the feature refinement module is as follows: the input features are enlarged by depthwise convolution, depthwise dilated convolution and point convolution to extract more local detail information; then, the intermediate features are fed into the coordinate attention module and the multi-head self-attention module through a fully connected layer to obtain the enhanced feature fusion output; finally, they are normalized by the LN layer and fed into the subsequent classification head and regression head to complete the detection and localization of the target.

4. The end-to-end visual language model for power inspection scenarios according to claim 1 or 2, characterized in that, The text branch includes a text prompt module and a text encoder. In the text prompt module, text prompt vectors are dynamically generated based on image categories. Contextual vocabulary is automatically modeled using learnable continuous vectors. These learnable text prompt vectors are composed of randomly initialized vectors and state text. When using a Transformer as the text encoder, a prompt vector with the same dimension as the text embedding needs to be generated. With the original embedding The elements are added bit by bit, then position encoding information is added, and finally the processed data is input into the Transformer encoder layer for calculation.

5. A defect detection system for power components in transmission lines, characterized in that, The invention includes an end-to-end visual language model for power inspection scenarios as described in any one of claims 1 to 4. The visual language model comprises an image branch and a text branch. The image branch extracts initial features from the image using a ResNet50 backbone network, then fuses multi-scale features through an improved feature pyramid, generates candidate regions using a region proposal network, and finally completes target detection and localization through region pooling, feature refinement modules, and classification and regression heads. The text branch first embeds the image category into different positions of the text prompt to construct a learnable text prompt, then obtains text features through a text encoder, and finally calculates the similarity between the text features and image region features for region classification.

6. The power device defect detection system in transmission lines according to claim 5, characterized in that, The improved feature pyramid includes a fusion module, which performs nearest neighbor interpolation upsampling on high-level features, performs 1×1 convolution operation on low-level features, and connects the upsampled high-level features with the 1×1 convolutioned low-level features to achieve the fusion of multi-scale features. Subsequently, the fused features are further enhanced in spatial resolution through subpixel convolution to generate a higher resolution feature map; finally, a 3×3 convolutional layer is used to refine the generated feature map.

7. The power device defect detection system in transmission lines according to claim 5 or 6, characterized in that, The specific operation process of the feature refinement module is as follows: the input features are enlarged by depthwise convolution, depthwise dilated convolution and point convolution to extract more local detail information; then, the intermediate features are fed into the coordinate attention module and the multi-head self-attention module through a fully connected layer to obtain the enhanced feature fusion output; finally, they are normalized by the LN layer and fed into the subsequent classification head and regression head to complete the detection and localization of the target.

8. The power device defect detection system in transmission lines according to claim 5 or 6, characterized in that, The text branch includes a text prompt module and a text encoder. In the text prompt module, text prompt vectors are dynamically generated based on image categories. Contextual vocabulary is automatically modeled using learnable continuous vectors. These learnable text prompt vectors are composed of randomly initialized vectors and state text. When using a Transformer as the text encoder, a prompt vector with the same dimension as the text embedding needs to be generated. With the original embedding The elements are added bit by bit, then position encoding information is added, and finally the processed data is input into the Transformer encoder layer for calculation.

9. A method for detecting defects in insulator strings, characterized in that, include: Images of insulator strings in transmission lines are acquired and used as input to a visual language model. Three status cues—"normal insulator," "damaged skirt," and "corroded steel cap"—are embedded in the text branch, and corresponding learnable text cues vectors are set for each. Multi-scale features of the insulator string images are extracted through the image branch to generate candidate regions. At the same time, the text branch dynamically generates learnable text cues based on preset categories and encodes them as text features. Finally, the similarity between the region features and the text features is calculated to complete the classification and localization of insulator defects.

10. A processing apparatus, characterized in that, It includes at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method as described in claim 9 by invoking the program instructions.

Images