Edge-assisted learning-based thermal infrared electrical equipment image semantic segmentation method

Abstract

The application discloses a thermal infrared electrical equipment image semantic segmentation method based on edge auxiliary learning, features extracted from an original input image are converted through a conversion module and a global information integration module, and edge detection and semantic segmentation tasks are simultaneously optimized, and meanwhile, a cross-guiding unit is used for simple feature transformation, two different branches in a decoder are interacted, and the two tasks achieve the effect of joint optimization. In addition, the application adds a true value supervision at the input end of the convolutional neural network module, so that the application can not only predict more accurate target boundaries, but also can obtain more accurate segmentation results through the edge detection branch to assist the semantic branch.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing technology, specifically relating to a semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning. Background Technology

[0002] Electrical equipment operating for extended periods is susceptible to malfunctions due to external environmental and weather factors, posing a threat to the safe operation of power systems. In recent years, with continuous technological advancements, infrared diagnostic technology has begun to be applied in fault diagnosis. This is because most faults, such as poor contact, insulation aging, or magnetic circuit faults, manifest as heat generation, and thermal infrared images can capture the equipment's temperature information. Currently, more research is focused on electrical equipment inspection based on thermal infrared images. However, target detection methods only provide a general target area; semantic segmentation methods can obtain more precise location information.

[0003] Image semantic segmentation aims to assign a corresponding semantic label to each pixel in an image, enabling a semantic reasoning process from coarse-grained to fine-grained, ultimately obtaining a segmented image with pixel semantic annotations. This paper studies the semantic segmentation of thermal infrared electrical equipment, with the goal of segmenting different electrical devices from a single thermal infrared image.

[0004] Accurately distinguishing electrical equipment from thermal infrared images is a highly challenging task. Infrared thermal imaging typically results in low image contrast and blurred object edges. Furthermore, electrical equipment often has a complex structure with sharp protrusions, making it difficult for networks to accurately identify object boundaries. Existing methods developed for natural scenes are also difficult to directly apply to thermal infrared images of electrical equipment.

[0005] Most existing semantic segmentation methods are improvements based on fully convolutional networks (FCNs). FCNs were pioneering applications of deep learning for semantic segmentation, establishing a general network model framework for image semantic segmentation. FCNs can accept images of arbitrary size as input and were the first end-to-end segmentation network model. Furthermore, for pixel-level classification tasks like semantic segmentation, using contextual information can capture long-range dependencies, improve the internal consistency of objects, and thus further enhance segmentation accuracy. Simultaneously, edge information is also crucial for accurate semantic segmentation.

[0006] However, the existing technical solutions mentioned above only consider using edge information to assist the segmentation task in a one-way manner, while ignoring the fact that edge detection and semantic segmentation are two complementary tasks. The clues obtained by interactively learning edge detection and semantic segmentation tasks have a positive impact on the final prediction. Summary of the Invention

[0007] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning. This invention uses a dual-path decoding structure to jointly perform interactive optimization learning on two different tasks: semantic segmentation and edge optimization, to solve the problem of multi-target segmentation in thermal infrared images of electrical equipment in existing technologies, thereby obtaining higher quality segmentation results.

[0008] Technical solution: The present invention provides a semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning, comprising the following steps:

[0009] Step S1: Input the image of electrical equipment with fine annotations into the neural network, where a ResNet-101 convolutional neural network pre-trained on the ImageNet dataset is used as the network feature encoding part;

[0010] Step S2: Use ResNet-101 convolutional neural network to extract four multi-level features C1 to C4 at different scales from the input image. Then, through the conversion module, the number of channels of the first three layers of features C1 to C3 in the network backbone is first unified to 256. The converted features are labeled as T1 to T3 respectively.

[0011] Step S3: The deepest feature C4 is fed into a global information integration module to obtain a feature containing global context information, which is labeled as T4;

[0012] Step S4: Perform edge prediction from T4 features to obtain edge features E4, and use the edge information as auxiliary learning information. Then, add an edge decoding branch to obtain edge features E1 to E4 with resolutions corresponding to semantic features at different scales.

[0013] Step S5: Use features T1 to T4 to form a decoding path, combine this decoding path branch with the edge decoding branch in step S4, and then use the cross-guided unit to jointly learn and optimize the semantic features and edge features.

[0014] Step S6: The output features of each cross-guided unit obtained in step S5 are concatenated with the corresponding semantic features along the channel and used as the input of the semantic feature part of the next cross-guided unit. These are denoted as features S1 to S3 in sequence. The input of the first cross-guided unit is the semantic feature S4 predicted from the T4 feature. The final output is the prediction map.

[0015] Step S7: Supervise the training of the network model by combining the cross-entropy loss function and the binary cross-entropy loss function.

[0016] This invention simultaneously optimizes edge detection and semantic segmentation tasks. To better achieve joint optimization of semantic segmentation and edge optimization, a cross-guidance unit is designed. This unit interacts two different branches in the decoder through simple feature transformation, enabling joint optimization of the two tasks. Furthermore, this invention adds ground truth supervision to the input of the convolutional neural network modules, allowing it to not only predict more accurate target boundaries but also assist the semantic branch in obtaining more accurate segmentation results through the edge detection branch.

[0017] Furthermore, in step S1, the ResNet-101 convolutional neural network removes the shared parameters of the fully connected layers. The feature encoder extracts four features from the input electrical equipment image, namely C1, C2, C3, and C4, with resolutions of 120×120, 60×60, 30×30, and 15×15, respectively.

[0018] Furthermore, the specific content of the global information integration module (which refines and enhances the feature map) in step S3 converting feature C4 into feature T4 is as follows:

[0019] First, feature C4 is input into the local information integration module. Then, feature C4′ with a global receptive field is obtained through pyramid pooling. At the same time, convolution operation is used to fine-tune feature C4 to obtain C4”. Next, the feature obtained by multiplying C4′ and C4” is added to feature C4 pixel by pixel to obtain the final output T4.

[0020] Furthermore, the specific workflow of the cross-guiding unit (which further refines and improves the segmentation results) in steps S5 and S6 is as follows:

[0021] Semantic features and edge features are obtained through feature T4, and then feature X is obtained by performing feature transformation on the semantic features and edge features. The i-th feature is represented as:

[0022]

[0023] Among them, S i E is the input corresponding to the i-th semantic feature map. iIt corresponds to the i-th edge feature map input α(E) i ), Dimensions and E i Same, indicating that for E i Different convolution operations were used respectively;

[0024] Then the acquired feature X i With S i The fusion process combines segmentation features and edge features for optimization. The final output D of the cross-guided unit is represented as follows:

[0025] D = Upsample(Sum(S) i ,X i ))

[0026] Meanwhile, semantic ground truth and edge ground truth are used at both ends of the input of each cross-guided unit to supervise the generated features to mutually constrain them, thereby optimizing the edge features and segmentation features to obtain more accurate segmentation results.

[0027] Furthermore, in step S7, the cross-entropy loss function and the binary cross-entropy loss function are combined to form L.

[0028]

[0029] The loss function combination L is a combination of the auxiliary loss of the two branches in the decoder and the final prediction loss, where n is the number of cross-guided units used, n=4, and auxiliary losses are added at both ends of each cross-guided unit;

[0030] l s The cross-entropy loss function applied to segmentation tasks is calculated as follows:

[0031]

[0032] Where H and W represent the height and width of the image, respectively, and S x,y The truth label for pixel (x, y) is given. If the category is k, then... Otherwise, it is 0; This represents the probability that a pixel at position (x, y) belongs to the k-th class.

[0033] Additionally, binary cross-entropy loss l is used in the marginal branches. e Function supervision, l e The calculation is as follows:

[0034]

[0035] Among them, E x,y P represents the truth label of pixel (x,y). x,yThis represents the probability of an edge object at position (x, y).

[0036] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0037] (1) This invention combines edge detection and semantic segmentation tasks, and adopts a decoding-encoding structure, while building a global information integration module in the encoder to improve the quality of the segmentation results.

[0038] (2) In order to better achieve the joint optimization of semantic segmentation and edge optimization, the present invention has designed a cross-guidance unit. The cross-guidance unit module interacts the two different branches in the decoder through simple feature transformation, so that the two tasks can achieve the effect of joint optimization.

[0039] In summary, this invention combines edge detection and semantic segmentation tasks, utilizes an interactive module to achieve mutual optimization, and employs a global information integration module to aggregate multi-scale information, thereby obtaining better segmentation performance and ultimately generating a high-quality prediction map. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0041] Figure 2 This is a schematic diagram of the segmentation network model of the present invention;

[0042] Figure 3 This is a schematic diagram of global information integration in the embodiment;

[0043] Figure 4 This is a schematic diagram of the cross-guide unit in the embodiment;

[0044] Figure 5 This is a visual comparison diagram for the embodiments. Detailed Implementation

[0045] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0046] like Figure 1 As shown, the semantic segmentation method for thermal infrared electrical equipment images using joint edge-assisted learning in this invention specifically includes the following steps:

[0047] Step S1: Input the image of electrical equipment with fine annotations into the neural network, where a ResNet-101 convolutional neural network pre-trained on the ImageNet dataset is used as the network feature encoding part;

[0048] Step S2: Use ResNet-101 convolutional neural network to extract four multi-level features C1 to C4 at different scales from the input image. Then, through the conversion module, the number of channels of the first three layers of features C1 to C3 in the network backbone is first unified to 256. The converted features are labeled as T1 to T3 respectively.

[0049] Step S3: The deepest feature C4 is fed into a global information integration module to obtain a feature containing global context information, which is labeled as T4;

[0050] Step S4: Perform edge prediction from T4 features to obtain edge features E4, and use the edge information as auxiliary learning information. Then, add an edge decoding branch to obtain edge features E1 to E4 with resolutions corresponding to semantic features at different scales.

[0051] Step S5: Use features T1 to T4 to form a decoding path, combine this decoding path branch with the edge decoding branch in step S4, and then use the cross-guided unit to jointly learn and optimize the semantic features and edge features.

[0052] Step S6: The output features of each cross-guided unit obtained in step S5 are concatenated with the corresponding semantic features along the channel and used as the input of the semantic feature part of the next cross-guided unit. These are denoted as features S1 to S3 in sequence. The input of the first cross-guided unit is the semantic feature S4 predicted from the T4 feature. The final output is the prediction map.

[0053] Step S7: Supervise the training of the network model by combining the cross-entropy loss function and the binary cross-entropy loss function.

[0054] This invention constructs an edge-assisted learning semantic segmentation network for semantic segmentation of electrical equipment in thermal infrared images. By combining edge detection and semantic segmentation tasks and utilizing a cross-guided unit to achieve mutual optimization, and by using a global information integration module to aggregate multi-scale information, better segmentation performance is obtained.

[0055] like Figure 2 As shown, the segmentation method in this embodiment is based on a dual-stream decoding cross-task interaction network. In this network model, the edge detection task and the semantic segmentation task interact, and an interaction module is used to achieve mutual optimization. Furthermore, a global information integration module that aggregates multi-scale information is used for processing.

[0056] Example 1:

[0057] This embodiment uses the E2TI-SO thermal infrared electrical equipment image dataset, with images captured by handheld infrared cameras in substation settings. E2TI-SO contains 4839 finely annotated thermal infrared images, covering 18 categories of equipment, including insulators, arresters, current transformers, potential transformers, disconnect switches, circuit breakers, and transformer bodies. The dataset is divided into 3226 training images and 1613 test images.

[0058] In this embodiment, the E2TI-SO training set is first randomly flipped, rotated, cropped, and subjected to affine transformations to increase the diversity of training samples. To ensure convergence of the detection model, the network in this embodiment is trained on an NVIDIA Tesla P100 PCIEP GPU platform with a training batch size of 14, and all training samples are trained 250 times. The backbone parameters of the network are determined by ResNet-101, while other convolutional layers are initialized using PyTorch's default settings. The network is trained using the Adams optimization method with a learning rate of 0.0001 and an input size of 480×480.

[0059] To facilitate quantitative evaluation, this embodiment uses the Mean Intersection over Union (MIoU) as the performance metric for the model. The ratio between the intersection and union of two sets represents the overlap between the ground truth labels and the predicted results, and is currently the most frequently used and common evaluation metric in the field of semantic segmentation. Furthermore, when calculating the MIoU, the background is considered as one of the categories.

[0060] The technical solution of this invention is compared with other existing technologies.

[0061] This embodiment compares the network of the technical solution of the present invention with 10 other methods.

[0062] The comparison methods include nine visible light-based image segmentation methods: FCN, Deeplabv3, PSPNet, CGNet, OCNet, DNLNet, DUNet, DANet, and EfficientFCN. Additionally, one existing infrared image segmentation method, ECCNN, is also included.

[0063] Quantitative comparison:

[0064] The specific comparative test results of this embodiment are shown in Table 1. Table 1 presents the performance comparison results of these methods on the E2TI-SO dataset, and details the Intersection over Union (IoU) results for each category. As can be seen from Table 1, this embodiment achieves the best results for 15 categories. Even if some categories do not reach the optimal results, the differences compared to other existing technologies are small.

[0065] Table 1

[0066]

[0067] The last row of Table 1 shows the average intersection-union (AUC) results for each method. The proposed solution outperforms the best visible light image segmentation method, DANet, by 3.08%, and the infrared image segmentation method, ECCN, by 3.81%. Overall, this embodiment significantly surpasses other methods.

[0068] Qualitative evaluation:

[0069] like Figure 5 As shown, this embodiment compares the final visual results with other technical solutions. This embodiment clearly demonstrates that the segmented targets have sharper and more accurate edges. For example, the blue and yellow targets in the first row have more precise edges. The beige target segmentation result in the eighth row almost lost some structure in the semantic segmentation methods for the three natural scene images, while this embodiment can obtain the approximate structure of that part.

Claims

1. A semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning, characterized in that: Includes the following steps: Step S1: Input the image of electrical equipment with fine annotations into the neural network, where a ResNet-101 convolutional neural network pre-trained on the ImageNet dataset is used as the network feature encoding part; Step S2: Use a ResNet-101 convolutional neural network to extract four multi-level features at different resolutions from the input image. And through the transformation module, the features of the first three layers in the network backbone are first transformed. The number of channels is unified to 256, and the converted feature labels are denoted as follows: ; Step S3: Extract the deepest features The data is fed into the global information integration module, resulting in a feature containing global context information, which is then labeled as... The specific method is as follows: First, the features The input is processed by the local information integration module, and then features with a global receptive field are obtained through pyramid pooling. Simultaneously using convolution operations on features Fine-tuning was performed to obtain Next, and Multiplying the feature obtained by multiplication and then adding the feature The final output is obtained by adding pixels one by one. ; Step S4, from Edge prediction is performed on the features to obtain edge features. Furthermore, edge information is used as auxiliary learning information, and then an edge decoding branch is added to obtain edge features at resolutions corresponding to semantic features at different scales. ; Step S5: Utilize features A decoding path is formed, and this decoding path branch is combined with the edge decoding branch in step S4. Then, the cross-guided unit is used to jointly learn and optimize the semantic features and edge features. Step S6: Concatenate the output features of each cross-guided unit obtained in step S5 with the corresponding semantic features along the channel, and use this as the input to the semantic feature part of the next cross-guided unit, denoted as features in sequence. The input of the first cross-guide unit is from Semantic features predicted from features The final output is the predicted image; The specific workflow of the cross-guided unit is as follows: Through features Semantic features and edge features are obtained through semantic prediction and edge prediction, respectively. Then, feature transformation is performed on the semantic features and edge features to obtain the final features. , No. Each feature is represented as: ， ； in, It is the corresponding number Input semantic feature maps, It is the corresponding number Input edge feature maps Dimensions and Same, indicating the same Different convolution operations were used respectively; Then the acquired features and The fusion process combines segmentation features and edge features for optimization. The final output D of the cross-guided unit is represented as: ； Meanwhile, semantic ground truth and edge ground truth are used at both ends of the input of each cross-guided unit to supervise the generated features to mutually constrain them, thereby optimizing the edge features and segmentation features to obtain more accurate segmentation results. Step S7: Supervise the training of the network model by combining the cross-entropy loss function and the binary cross-entropy loss function.

2. The semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning according to claim 1, characterized in that: In step S1, the ResNet-101 convolutional neural network removes the shared parameters of the fully connected layers. The feature encoder in the convolutional neural network extracts four features from the input electrical equipment image, which are as follows: The resolutions of these four features are as follows: , , and .

3. The semantic segmentation method for thermal infrared electrical equipment images based on edge-assisted learning according to claim 1, characterized in that: In step S7, the cross-entropy loss function and the binary cross-entropy loss function are combined as follows: , ； Loss function combination The auxiliary loss of the two branches in the decoder is combined with the final prediction loss, where, This refers to the number of cross-boot units used. At the same time, auxiliary losses are added at both ends of each cross-guided unit; The cross-entropy loss function applied to segmentation tasks is calculated as follows: ； in, These represent the height and width of the image, respectively. Represents pixels The truth label, if the category is ,but Otherwise, it is 0; Indicates the location The upper pixel belongs to the first The corresponding probability of the class; Additionally, binary cross-entropy loss is used in the marginal branches. Functions are used for supervision. The calculation is as follows: ； in, Represents pixels truth labels, Indicates the location The corresponding probability of the upper edge object.

Images