An infrared image power device interactive segmentation method and system

Abstract

The application provides an infrared image power equipment interactive segmentation method and system, acquires a plurality of known pixel-level labels of power equipment infrared images, the pixel-level labels are used as real masks to constitute a power equipment segmentation data set; a large image segmentation model is constructed, and pre-training is performed on the large image segmentation model; input information of a user is simulated by using an interactive click generation strategy by using the known real masks; the infrared image and the simulated input information are jointly input into the trained large image segmentation model to obtain a predicted mask of the power equipment and obtain a segmentation result. The application can obtain accurate segmentation results by using a small amount of click interaction, can also be used for quickly constructing a large-scale infrared image data set, and promotes the development of infrared image automatic segmentation and defect diagnosis.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and more specifically, to an interactive segmentation method and system for infrared image power equipment. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the expansion and increasing complexity of power systems, the health monitoring of power equipment has become increasingly important, serving as a crucial guarantee for modern industrial production and daily life. In today's information age, intelligent power equipment monitoring has become a trend. This involves real-time monitoring of the operating status of power equipment to diagnose and analyze potential faults and provide automatic early warnings, thereby achieving efficient and comprehensive management and maintenance. Intelligent monitoring of power equipment is of great significance, not only improving the reliability and safety of equipment operation but also saving substantial human resources and enhancing the overall operation and management level of the power system.

[0004] In recent years, monitoring technologies for power equipment have been continuously enriched and developed. Among numerous technical means, infrared imaging technology, as a non-contact detection method, can quickly and accurately detect temperature distribution, thermal damage, and structural defects in power equipment, and can operate around the clock and in all weather conditions, thus gaining widespread application. To improve the intelligence level of power equipment condition monitoring, intelligent image processing and analysis methods, including automatic segmentation of power equipment from infrared images and automatic fault diagnosis of power equipment, have received widespread attention. Among these, power equipment segmentation is the foundation for subsequent fault analysis and early warning.

[0005] Current infrared image segmentation methods for power equipment are mostly based on deep learning technology. To effectively train deep segmentation models, a large number of real-world scene images and their corresponding segmentation masks are needed as training data. However, according to the inventors, the acquisition of infrared image segmentation masks for power equipment currently relies mainly on manual annotation. This method not only consumes a significant amount of time and human resources but also requires annotators to possess specialized knowledge of power equipment, making it difficult to improve the scale and quality of the annotated data. This significantly hinders the improvement of the intelligent level of power equipment condition monitoring.

[0006] In recent years, with the rapid development of deep learning technology, image segmentation models based on deep convolutional neural networks, such as DeepLab and HRNet, and models based on Transformer network structures, such as SegFormer, have gradually been applied to the segmentation of electrical equipment in infrared images. For example, the work of Liu et al., "Deep Residual UNet Network-based Infrared Image Segmentation Method for Electrical Equipment," and the work of Zhao et al., "Infrared Image Segmentation Method of Current Transformer Based on DeepLabv3+ Network," respectively utilize UNet and DeepLabv3+ networks to segment electrical equipment in infrared images. These methods typically transfer networks trained on visible light image datasets to their own labeled small-scale infrared image datasets, adapting them to infrared image and electrical equipment segmentation tasks through model optimization. However, due to the time-consuming and labor-intensive labeling process, self-constructed infrared image datasets are usually small in scale, which limits the performance of the models to some extent. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides an interactive segmentation method and system for infrared images of power equipment. This invention leverages the powerful generalization performance of the Segment Anything Model (SAM), a large-scale image segmentation model trained on a massive visible light dataset, to perform interactive segmentation of infrared images. On one hand, it achieves accurate segmentation results with minimal user interaction; on the other hand, it can be used to rapidly construct large-scale infrared image datasets, promoting the development of automatic infrared image segmentation and defect diagnosis.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] An interactive segmentation method for infrared image power equipment includes the following steps:

[0010] Infrared images of power equipment with multiple known pixel-level labels are acquired, and these pixel-level labels are used as ground masks to form a power equipment segmentation dataset.

[0011] Obtain a large-scale image segmentation model pre-trained in a large number of visible light images;

[0012] Using known real masks, an interactive click generation strategy is employed to simulate user input.

[0013] The infrared image and the simulated input information are input together into the trained image segmentation model to obtain the predicted mask of the power equipment and the segmentation result.

[0014] As an alternative implementation, the method further includes calculating the intersection-union ratio (IU) of the mask and the real mask, interactively adding input information to improve the prediction results until a preset IU value is reached, and recording the number of times the input information is added.

[0015] As an alternative implementation, the large-scale image segmentation model includes an image encoder, a cue encoder, and a lightweight decoder. The infrared image is processed by the image encoder to extract image features, and the user's click is used as a sparse cue to extract cue features through the cue encoder. Both features are input into the lightweight decoder to obtain the predicted mask of the target of interest.

[0016] As an alternative implementation, during the training of the large image segmentation model, the output of the previous round of interaction is also incorporated into the image features for the next round of prediction.

[0017] As a further step, the image encoder comprises a ViT / 16 network with 14×14 window attention and four global attention blocks, and the input image resolution used by the image encoder is obtained by scaling the original image and padding the short side.

[0018] As a further step, the cue encoder includes sparse cue, whereby for sparse cue represented by a click, two-dimensional spatial coordinates are used to represent the position information of the click in the image, and 1 and 0 are used to represent positive and negative click types, respectively.

[0019] Each click is mapped to a multidimensional feature vector containing location encoding and type information.

[0020] As a further step, the cue encoder contains dense cues represented by masks, encoded using convolutional layers, taking a previous mask with a resolution lower than the input image as input, and then using two convolutional layers with different output channels to further reduce the feature scale;

[0021] A 1×1 convolutional layer is used to map the channel dimension to a set dimension. Each convolutional layer is followed by GELU activation and layer normalization. The mask features obtained after convolution have the same scale as the image features output by the image encoder. The mask features are then fused with the pre-extracted image features by element-wise summation.

[0022] As a further step, the lightweight decoder comprises two decoder layers and a mask prediction head, each decoder layer being used for self-attention of cue features, cross-attention from cue features to image features, a multilayer perceptron, and cross-attention from image features to cue features.

[0023] The second decoder layer obtains the updated image features and prompt features as input from the first decoder layer;

[0024] After obtaining the fused features through two decoder layers, the output mask is obtained using a mask prediction head.

[0025] As a further step, positional encoding is added to the image features before they enter the attention layer; when the cue features containing positional encoding pass through the attention layer, residual connections are used to re-add them to the updated features, preserving the location and type information of the click as much as possible.

[0026] As an alternative implementation, the mask prediction head uses two transposed convolutional layers to upsample the fused features by a factor of 4;

[0027] The fused features are cross-attentioned again with the image features, and the updated output features are passed to the multilayer perceptron. The feature vector output by the multilayer perceptron has the same dimension as the upsampled feature.

[0028] The output of the multilayer perceptron is multiplied point-by-point in space with the upsampled features to obtain the output prediction mask.

[0029] An interactive segmentation system for power equipment based on infrared images, comprising:

[0030] The dataset construction module is configured to acquire infrared images of power equipment with multiple known pixel-level labels, which serve as ground truth masks to form a power equipment segmentation dataset.

[0031] The model building and training module is configured to build a large image segmentation model and pre-train it.

[0032] The simulation module is configured to use an interactive click generation strategy to simulate user input information using a known real mask.

[0033] The interactive segmentation module is configured to input both infrared images and simulated input information into a trained image segmentation model to obtain a predicted mask for the power equipment and thus the segmentation result.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] This invention proposes an interactive segmentation method for power equipment in infrared images based on a pre-trained large-scale model (SAM). This method allows users to iteratively control the segmentation results through interactive clicks until a satisfactory segmentation effect is achieved. It helps annotators quickly and accurately label any power equipment target component of interest in infrared images, significantly saving manpower and time costs while rapidly building a large-scale dataset for training automatic infrared image segmentation and fault diagnosis models.

[0036] This invention eliminates the need to re-extract image features after each click. It employs a post-fusion method, separating the image extraction process from the click interaction process. This allows each image to have its features extracted and saved only once by an image encoder, ready for use in subsequent rounds of interaction. This design reduces redundant computation and ensures that the model can process each user prompt in real-time, even when using a large-scale image encoder, achieving a balance between segmentation performance and running speed. Attached Figure Description

[0037] The accompanying drawings, which constitute a part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute a limitation thereof.

[0038] Figure 1 This is a flowchart of an example of the present invention.

[0039] Figure 2 This is a schematic diagram of the SAM interactive segmentation framework in an example of the present invention.

[0040] Figure 3 This is a structural diagram of the lightweight encoder in an example of the present invention.

[0041] Figure 4 This is a segmentation effect diagram of some device examples in the embodiments of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] Unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0045] Example 1

[0046] An interactive segmentation method for infrared image power equipment includes the following steps:

[0047] (1) Obtain a series of infrared images of power equipment with known pixel-level labels (i.e., real masks) to form a power equipment segmentation dataset;

[0048] (2) Construct an interactive segmentation framework for infrared images of power equipment based on a large SAM pre-trained model, including three modules: image encoder, cue encoder and lightweight decoder. In this embodiment, PyTorch is used to construct the segmentation framework.

[0049] The constructed interactive segmentation framework is based on a large pre-trained SAM model, which mainly consists of three modules: an image encoder, a cue encoder, and a lightweight decoder. Specifically, the infrared image is processed by the image encoder to extract image features, and user clicks are used as sparse cues to extract cue features through the cue encoder. Both types of features are then input into the lightweight decoder to obtain the predicted mask of the target of interest. Furthermore, the annotation process often requires multiple rounds of user interaction; therefore, this model also incorporates the output probability map of the previous round of interaction into the image features for the next round of prediction. This approach preserves previous segmentation information, thereby achieving better prediction results.

[0050] User clicks in this article mainly refer to clicks made by users on foreground or background areas in an image.

[0051] The image encoder module described above is used to extract infrared image features. In general, the image encoder can be any network that outputs C×H×W image features.

[0052] To leverage the powerful pre-trained model, this embodiment uses the Vision Transformer (ViT) as the image encoder, with minor modifications to enable it to handle higher resolution inputs.

[0053] Specifically, this embodiment uses a ViT / 16 network with 14×14 window attention and four global attention blocks. The output size of the image encoder is reduced by a factor of 16 compared to the input image. The input image used by the encoder has a resolution of 1024×1024, which is obtained by scaling the original image and padding the shorter side. The output feature size of the encoder is 64×64×256.

[0054] This embodiment provides three encoder versions of different sizes, referred to as ViT-B, ViT-L and ViT-H respectively, from smallest to largest. As the encoder size increases, its generalization performance and segmentation accuracy also improve step by step.

[0055] The aforementioned prompt encoder module maps prompt information into feature vectors. To correspond to the masks generated during user clicks and interactions, this embodiment considers two prompt methods: sparse prompts and dense prompts.

[0056] For sparse cues represented by clicks, two-dimensional spatial coordinates are used to represent the location information of the click in the image, with 1 and 0 representing positive and negative click types, respectively (where positive clicks are clicks in the foreground target area and negative clicks are clicks in the background area). Then, each click is mapped to a 256-dimensional feature vector containing both location encoding and type information.

[0057] For dense cues represented by masks, convolutional layers are used for encoding. A previous mask with a resolution four times lower than the input image is taken as input, and then two 2×2 convolutional layers with 4 and 16 output channels are used to further reduce the feature scale by a factor of four. Finally, a 1×1 convolution maps the channel dimensions to 256 dimensions. Each convolutional layer is followed by GELU activation and layer normalization. The resulting mask features have the same scale as the image features output by the image encoder.

[0058] Since masks and images have a spatial correspondence, mask features are fused with pre-extracted image features through element-wise summation. It's important to note that in the first round of interactive segmentation, since there are no prior masks, a learnable feature vector representing "no mask" needs to be summed element-wise with the image features.

[0059] The lightweight decoder module is used to map image features and cue features to an output mask. This module consists of two decoder layers and a mask prediction head.

[0060] The decoder layer is derived from the decoder module in the Transformer. Each decoder layer performs four steps: self-attention of cue features, cross-attention from cue features to image features, multi-layer perceptron (MLP), and cross-attention from image features to cue features.

[0061] During cross-attention, the 64×64×256 image features are treated as 64×64 256-dimensional vectors. Each self-attention, cross-attention, and multilayer perceptron module has a layer normalization and a Dropout of 0.1. The model stacks these modules in two layers, with the second layer taking the updated image features and cue features from the first layer as input.

[0062] It's important to note that, to ensure the decoder can capture crucial geometric information, we add positional encoding to image features before they enter the attention layer. Furthermore, when the cue features containing positional encoding pass through the attention layer, residual connections are used to re-add them to the updated features, thus preserving the click's location and type information as much as possible.

[0063] After obtaining the fused features through two decoder layers, this embodiment employs a simple mask prediction head to acquire the output mask. On one hand, the fused features are upsampled by a factor of 4 using two transposed convolutional layers. On the other hand, the fused features are again cross-attentioned with the image features, and the updated output features are passed to an MLP whose output feature vector dimension is consistent with the upsampled feature dimension. Then, the MLP output and the upsampled features are multiplied point-by-point in space to obtain the output predicted mask.

[0064] The two transposed convolutional kernels used to upsample the output features are 2×2 in size, with a stride of 2, and output channel sizes of 64 and 32, respectively. GELU activation and layer normalization are performed after each convolutional layer.

[0065] (3) Using known real masks, an interactive click generation strategy is used to simulate user input.

[0066] In this embodiment, let O be the set of pixels in the actual mask of the object. To simulate a user's positive click, we randomly select n pixels from O as positive clicks, and these clicks must meet the following filtering conditions: any two clicks must be at least dstep pixels apart; and any click must be at least dmargin pixels away from the target boundary. Furthermore, several negative clicks are generated in the background mask using a similar strategy.

[0067] (4) Input the infrared image and the simulated user clicks into the trained infrared image power equipment interactive segmentation framework to obtain the prediction mask of the power equipment.

[0068] In the actual implementation, the host CPU used was an Intel Xeon E5-2620 v4, and the GPU was configured as a GeForce GTX1080Ti.

[0069] (5) Calculate the Intersection over Union (IoU) ratio between the predicted mask and the real mask, improve the prediction results by interactively adding clicks until the preset IoU value is reached, and record the number of clicks added.

[0070] This embodiment collected 327 infrared images containing power equipment as test data, and pixel-level mask annotations were applied to each image to evaluate the interactive segmentation performance of the model. The dataset includes four types of power equipment components: circuit breakers, surge arresters, current transformers, and voltage transformers. It should be noted that for some difficult samples, even with continuous clicks, it is difficult to achieve a high IoU. Therefore, to avoid the testing process from entering an infinite loop, the maximum number of clicks was set to 20 in this embodiment.

[0071] Two standard evaluation metrics are used in interactive segmentation tasks: Number of Clicks (NoC) and Number of Failures (NoF). NoC@p% represents the average number of clicks required to achieve p% IoU, and NoF@p% represents the number of samples that still cannot achieve p% IoU after reaching the maximum number of clicks (20 in this example). The smaller the values ​​of NoC and NoF, the better the performance of the interactive segmentation model.

[0072] The three versions of the SAM model, ViT-B, ViT-L and ViT-H, were tested on a self-collected infrared power equipment dataset, and the evaluation results are shown in Table 1.

[0073] Table 1 Evaluation Indicators

[0074]

[0075] As shown in the table above, due to SAM's strong generalization ability, it can still achieve interactive segmentation of power equipment in infrared images even without specific fine-tuning on the infrared dataset. The performance of the three model versions on the power scene infrared dataset is similar, with an average of only 5.34 clicks required to achieve a segmentation intersection-over-union (IoU) ratio of over 80%, while the average number of clicks to achieve an IoU ratio of over 90% is a maximum of 10.24. In practical applications, to save computational resources and improve the speed of single-icon annotation, the smallest version, ViT-B, can be used.

[0076] The experimental results show that the method provided in this embodiment can help annotators accurately annotate power equipment from infrared images with a small number of clicks, demonstrating the practicality of this method.

[0077] also, Figure 4 The results show the segmentation results of some samples in the test set, including the four categories of electrical equipment in the dataset. These experimental results demonstrate that the model used in this method can accurately segment the target area of ​​the equipment and adapt to different temperature scenarios and complex background environments, exhibiting good robustness.

[0078] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An interactive segmentation method for power equipment using infrared images, characterized in that, It includes the following steps: Infrared images of power equipment with multiple known pixel-level labels are acquired, and these pixel-level labels are used as ground masks to form a power equipment segmentation dataset. A large-scale image segmentation model is constructed and pre-trained. The large-scale image segmentation model includes an image encoder, a cue encoder, and a lightweight decoder. The infrared image is processed by the image encoder to extract image features, and the user's click is used as a sparse cue to extract cue features through the cue encoder. The two types of features are input into the lightweight decoder to obtain the prediction mask of the target of interest. During the training of the large-scale image segmentation model, the output of the previous round of interaction is also fused into the image features for the next round of prediction. Using known real masks, an interactive click generation strategy is employed to simulate user input. The interactive click generation strategy is as follows: Let O be the set of real mask pixels of an object. Simulate the user's forward click and randomly select n pixels in O as the forward click. At the same time, the forward clicks satisfy the following filtering conditions: any two clicks must be at least dstep pixels apart; and any click must be at least dmargin pixels away from the target boundary. The infrared image and the simulated input information are input together into the trained image segmentation model to obtain the predicted mask of the power equipment and the segmentation result.

2. The interactive segmentation method for infrared image power equipment as described in claim 1, characterized in that, It also includes calculating the intersection-union ratio (IUR) of the mask and the real mask, and improving the prediction results by interactively adding input information until a preset IUR value is reached, recording the number of times input information is added.

3. The interactive segmentation method for infrared image power equipment as described in claim 1, characterized in that, The image encoder described herein comprises a ViT / 16 network with 14×14 window attention and four global attention blocks. The input image resolution used by the image encoder is obtained by scaling the original image and padding the short side.

4. The interactive segmentation method for infrared image power equipment as described in claim 1, characterized in that, The prompt encoder includes sparse prompts. For sparse prompts represented by clicks, two-dimensional spatial coordinates are used to represent the position information of the click in the image, and 1 and 0 are used to represent positive and negative click types, respectively. Each click is mapped to a multidimensional feature vector containing location encoding and type information; Alternatively, the cue encoder contains dense cues represented by masks, encoded using convolutional layers, taking a previous mask with a resolution lower than the input image as input, and then using two convolutional layers with different output channels to further reduce the feature scale; A 1×1 convolutional layer is used to map the channel dimension to a set dimension. Each convolutional layer is followed by GELU activation and layer normalization. The mask features obtained after convolution have the same scale as the image features output by the image encoder. The mask features are then fused with the pre-extracted image features by element-wise summation.

5. The interactive segmentation method for infrared image power equipment as described in claim 1, characterized in that, The lightweight decoder comprises two decoder layers and a mask prediction head. Each decoder layer is used for self-attention of cue features, cross-attention from cue features to image features, multilayer perceptron, and cross-attention from image features to cue features. The second decoder layer obtains the updated image features and prompt features as input from the first decoder layer; After obtaining the fused features through two decoder layers, the output mask is obtained using a mask prediction head.

6. The interactive segmentation method for infrared image power equipment as described in claim 5, characterized in that, Before image features enter the attention layer, location encoding is added to them; when the cue features containing location encoding pass through the attention layer, residual connections are used to add them back to the updated features to preserve the location and type information of the click as much as possible.

7. An interactive segmentation method for infrared image power equipment as described in claim 5 or 6, characterized in that, The mask prediction head uses two transposed convolutional layers to upsample the fused features by a factor of 4. The fused features are cross-attentioned again with the image features, and the updated output features are passed to the multilayer perceptron. The feature vector output by the multilayer perceptron has the same dimension as the upsampled feature. The output of the multilayer perceptron is multiplied point-by-point in space with the upsampled features to obtain the output prediction mask.

8. An interactive segmentation system for power equipment based on infrared images, characterized in that, include: The dataset construction module is configured to acquire infrared images of power equipment with multiple known pixel-level labels, which serve as ground truth masks to form a power equipment segmentation dataset. The model building and training module is configured to build a large image segmentation model and pre-train it. The large image segmentation model includes an image encoder, a cue encoder, and a lightweight decoder. The infrared image is processed by the image encoder to extract image features, and the user's click is used as a sparse cue to extract cue features through the cue encoder. The two types of features are input into the lightweight decoder to obtain the prediction mask of the target of interest. During the training of the large image segmentation model, the output of the previous round of interaction is also fused into the image features for the next round of prediction. The simulation module is configured to use an interactive click generation strategy to simulate user input information using a known real mask. Let O be the set of real mask pixels of an object. To simulate a user's forward click, n pixels in O are randomly selected as forward clicks. At the same time, the forward clicks satisfy the following filtering conditions: any two clicks are at least dstep pixels apart; and any click is at least dmargin pixels away from the target boundary. The interactive segmentation module is configured to input both infrared images and simulated input information into a trained image segmentation model to obtain a predicted mask for the power equipment and thus the segmentation result.

Images