A semantic segmentation method for power accessories of transmission lines

Abstract

The present invention provides a method for semantic segmentation of power accessories of a transmission line, comprising: (1) using a drone equipped with an infrared thermal imager and a visible light camera to circle around a power pole tower, capturing infrared and visible light image pairs of power accessories, and obtaining an infrared and visible light original sample set; (2) using an edge intensity-based image registration module to process the original sample set, and generating a high-matching infrared and visible light image registration result; (3) obtaining an infrared and visible light image power accessories sample set; (4) performing final semantic segmentation on the labeled infrared and visible light images; (5) randomly dividing the infrared and visible light image sample set in (3) into a training set and a test set at a ratio of 4:1, and performing training to obtain a more accurate segmentation effect. The present invention provides a method for semantic segmentation of power accessories of a transmission line, which can accurately identify power accessories such as power pole towers, insulators, tension clamps, and shock-absorbing hammers in a transmission line.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic segmentation of power accessories, and in particular relates to a semantic segmentation method for power accessories of power transmission lines using the collaboration of infrared and visible light images. Background Art

[0002] Power component recognition is an important foundation for achieving functions such as power transmission line scene understanding, autonomous flight of inspection drones, and intelligent detection of equipment defects, and has significant research and practical significance. The key step in power component recognition is to distinguish the power components from the background and classify them by combining the extracted image semantic information with the power component differentiation results. Semantic segmentation methods can assign semantic category labels to all pixels in an image. When training deep learning-based semantic segmentation methods using images of power transmission line power components, it is possible to locate the position of different power components in the image and identify the type of equipment. This allows for fault diagnosis solutions to be determined based on the type of power component, significantly improving the intelligence of drone inspections.

[0003] However, semantic segmentation methods that rely solely on visible light images are easily disturbed by the rapid movement of drones and camera shake. At the same time, visible light imaging is affected by external environments such as lighting and weather. The reduction in visible light image quality will further lead to a decline in the performance of semantic segmentation algorithms. Therefore, how to utilize the temperature perception and susceptibility to external environmental interference of infrared images while eliminating redundant information between modalities has become the key to achieving semantic segmentation in all-weather power scenarios and thermal fault diagnosis of power accessories.

[0004] To address these issues, patent CN114612659A discloses a method and system for segmenting power accessories based on fused modal contrastive learning. This method uses a fused modal feature encoder to extract features from infrared and visible light image pairs, fuses them, and feeds them back into a feature extraction network to supervise specific modal features. Finally, the fused modal results are fed into a decoder and contrastive learning module to obtain the final fusion result. However, using infrared and visible light image pairs can lead to pixel mismatches between the infrared and visible light images due to subtle differences in registration, which in turn affects segmentation accuracy. Furthermore, fusing the infrared and visible light images equally ignores the specific properties of the modalities themselves, reducing modal fusion efficiency and affecting segmentation results. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a semantic segmentation method for power accessories of power transmission lines based on the collaboration of infrared and visible light images. During the feature extraction process of infrared and visible light images, the method adaptively selects supplementary information from infrared features and integrates it into visible light images, thereby accurately completing category mapping for all pixels of the image during the decoding process, overcoming the influence of the external environment and the rapid movement of drones, and accurately identifying power accessories such as power poles, insulators, tension clamps, and shock absorbers in transmission lines.

[0006] The present invention specifically provides a method for semantic segmentation of power accessories of power transmission lines using infrared and visible light images. The method comprises the following steps:

[0007] Step (1): Use a drone equipped with an infrared thermal imager and a visible light camera to circle the power tower and capture infrared and visible light image pairs of power accessories to obtain an infrared and visible light original sample set containing horizontal insulators, vertical insulators, insulation clamps, shock absorbers, and tower targets;

[0008] Step (2): using an edge intensity-based image registration module to process the infrared and visible light original sample set to generate an infrared and visible light image registration result with a high matching degree;

[0009] Step (3): Send the registered infrared and visible light image pairs to the sample set construction module to construct the infrared and visible light power accessories sample set. Use the labelme annotation tool to manually calibrate all visible light and corresponding infrared images for the five types of targets in step (1). The remaining unlabeled areas are used as the background to obtain the infrared and visible light image power accessories sample set.

[0010] Step (4): The annotated infrared and visible light image pairs are input into the feature fusion module of multi-level modal information feature extraction and weight adaptive learning to fully extract the specific infrared and visible light data features and fully fuse them by utilizing their complementarity. The fusion results are then sent to the high-level feature activation module to enhance the differential features and eliminate the influence of interference noise. Finally, the global feature map is sent to the multi-level decoding module to obtain the final semantic segmentation result.

[0011] Step (5): The infrared and visible light image sample sets in step (3) are randomly divided into training sets and test sets in a ratio of 4:1, and the training set is used to train the semantic segmentation network. The network parameters are updated with the cross entropy loss function as supervision to obtain more accurate segmentation results.

[0012] In step (2), the edge strength-based image registration module includes an edge strength detection module, a feature selection module, and a descriptor matching module, wherein the edge strength detection module includes the following steps:

[0013] (1) The original infrared and visible light image pair is input to extract the edge features of the infrared image. The infrared image A is eroded by the structuring element B as follows:

[0014]

[0015] Among them B z is the translation of vector z with respect to B, specifically E is the image after corrosion; similarly, the infrared image A is expanded by the structuring element B as follows:

[0016]

[0017] Among them B s ={x∈F|-x∈B}, where F is the expanded image. The process can be regarded as rotating the structuring element B 180 degrees around the origin and then corroding the infrared image A.

[0018] (2) According to (1), the inner edge D1 = A-(A!B) and the outer edge of the infrared image can be obtained Then obtain the basic gradient edge of the infrared image

[0019]

[0020] The feature selection module performs feature selection based on the ORB (Oriented FAST and rotated BRIEF) algorithm, and includes the following steps:

[0021] (1) Take a pixel block p of size S×S and obtain the key point q=(x,y) on the pixel block through smoothing. T The gray value p(q) at , and define the binary judgment standard Γ:

[0022]

[0023] where r = (x′, y′) T is the selected threshold key point, p(r) is the grayscale value at r;

[0024] (2) Select n pairs of position coordinates (x i ,y i ), and compare them according to the binary judgment standard Γ to obtain the n-dimensional feature descriptor f n (p):

[0025]

[0026] (3) A 2×n-order matrix M is transformed into a rotation matrix R θ Perform the operation to obtain the new matrix M θ for:

[0027]

[0028] in The rotation matrix R corresponding to the main direction θ of the key point q θ Expressed as

[0029] (4) From (3), the matrix M becomes a directed form, that is, the feature descriptor has rotation invariance, and the binary descriptor is:

[0030] g n (p,θ)=f n (p)|(x i ,y i )∈S θ .

[0031] The descriptor matching module comprises the following steps:

[0032] (1) Using the Hamming distance as the similarity measure of two points, the brute force matching method is used to roughly match the feature points;

[0033] (2) Solve the transformation matrix by the progressively consistent sampling method;

[0034] (3) Using the RANSAC algorithm to calculate the final transformation matrix H in the high-precision interior point set, and transforming the image according to the transformation matrix H to find the common area between the two images;

[0035] (4) Affine transformation is used as the solution model for registration, and its mathematical expression is:

[0036]

[0037] Where (x1, y1) represents the original coordinates, (x2, y2) represents the coordinates after projection transformation, H1 represents the affine transformation matrix, (a 02 ,a 12 ) indicates the offset.

[0038] Step (4) includes a sample set construction module, a multi-level modal information feature extraction module, a feature fusion module for weight adaptive learning, a high-level feature activation module, a multi-level decoding module, and a network training module, and includes the following steps:

[0039] (1) The annotated infrared image and visible light image pairs are input into two parallel encoders of corresponding modalities respectively. The two parallel encoders use ResNet50 as the backbone network, and the lower branch encoder changes the number of input channels of the first convolution layer to 1 to match the corresponding infrared grayscale image. The upper branch is used to extract the corresponding visible light image features;

[0040] (2) Starting from the second layer, the infrared image is fed into the weight adaptive learning feature fusion module located before each convolutional layer of the visible light encoder, and the fusion result is used as the input of the next convolutional layer of the visible light encoder to achieve auxiliary supervision of the infrared image on the visible light image;

[0041] (3) sending the final fusion result of (2) to the high-level feature activation module to enhance the difference features and eliminate the influence of interference noise, and obtain a global feature map;

[0042] (4) Send the global feature map to the multi-level decoding module to obtain the final semantic segmentation result.

[0043] The feature fusion module of the weight adaptive learning includes the following steps:

[0044] (1) Let the input visible light image feature be R i-1 , the infrared image feature is T i-1 , R i-1 With T i-1 After the feature connection, it is sent to a 3×3 convolution layer to reduce the number of channels to 1 / 4 of the original, denoted as P i ;

[0045] (2) Use four maximum pooling layers with sizes of 1, 5, 9, and 13 to process P i , we get four feature maps of different types and target sizes, and then send them to two convolutional layers, each containing a 3×3 convolution kernel and a ReLU activation function, to get four feature maps v j ,j=1,2,3,4;

[0046] (3) Upsample the result of (2) until it matches P i Keep consistent to form the residual structure, and P i After the number of channels is reduced to 2 and the up-sampled v j Perform pixel-level summation operation to fuse and obtain the fusion image U;

[0047] (4) Input the result in (3) into two 3×3 convolutional layers and activate the softmax function to obtain the dual-channel mapping W i ;

[0048] (5) Separate the result in (4) into two weight maps W i RGB and W i T And respectively with the input R i-1 With T i-1 Perform pixel-level multiplication to obtain the input of the next visible light encoder convolution layer

[0049]

[0050] (6) In each intermediate stage of the feature extraction encoder, a feature fusion module with weight adaptive learning is used to integrate the infrared image features into the visible light image features, and the results are sent to the next multi-level modal information feature extraction module for feature extraction. Steps (1) to (5) are continuously iterated until the final fusion result R5 is obtained.

[0051] The high-level feature activation module includes the following steps:

[0052] (1) Split R5 into two independent feature descriptors M avg With M max , and respectively M avg With M max It is fed into the adaptive average pooling layer and the adaptive maximum pooling layer to obtain an output result of size 1×1, where

[0053] M avg =AvgPool1(R5),M max =MaxPool1(R5);

[0054] (2) The two descriptors in (1) are sent to the shared block, which is implemented by two convolutional layers and a ReLU function. The size of the two convolutional layers is 1×1. The first convolutional layer reduces the number of channels to half of the original number. After the ReLU function, the second convolutional layer restores it. Then, the channel attention feature map is generated by element summation, sigmoid function and element multiplication:

[0055]

[0056] (3) The result of (2) F c It is sent to the Atrous Spatial Pyramid Pooling (ASPP) module to expand the feature receptive field and capture multi-scale information. After connecting the upsampling results, an adaptive global feature map G is obtained, and a 1×1 convolution layer is used to derive its channel number.

[0057] The multi-level decoding module comprises the following steps:

[0058] (1) Taking the first decoder as an example, the global feature map G, the corresponding visible light image encoder output feature C and the output O of the previous decoding module are fed into the decoder. Since the first decoder does not have the output of the previous decoding module, the global feature map G is used instead;

[0059] (2) In this example, C is R4, and in the other modules, they are R3 and R2 respectively. The output O of the previous decoding module and the output feature C of the visible light image encoder are fed into the adaptive spatial channel attention module to enhance its modality-specific features;

[0060] (3) Upsample the result in (2) to a resolution consistent with feature C, and reduce the number of channels of the three feature maps to 1 / 4 of the number of channels of feature map O;

[0061] (4) Perform pixel-level summation on the three results in (3) and reconstruct the summed feature map using a 3×3 convolution kernel;

[0062] (5) Repeat the above steps until the last decoder, which is fed into a 1×1 convolution kernel and then upsampled to change its number of channels to the number of segmentation target categories, and the final semantic segmentation result Y is obtained.

[0063] In step (5), the annotated infrared and visible light image groups are randomly divided into training and test sets in a ratio of 4:1, where the sizes of infrared and visible light images are both 640×480. The above network is trained using the training set. Given the prediction results Y = {Y(i), i = 1, 2, ..., m} and the true value image Y label ={Y label (i), i=1,2,...,m}, and use the cross entropy loss function as supervision to update the network parameters to obtain better segmentation effect.

[0064] Compared with the existing technology, the beneficial effects are: the semantic segmentation method of power accessories of transmission lines adds an infrared and visible light image registration module before performing the semantic segmentation task, and uses edge information to guide, which significantly improves the accuracy of image registration and reduces the degree of distortion after affine transformation; the method adds a multi-level feature extraction module to the power accessories semantic segmentation module, fully extracts specific infrared and visible light data features, constructs an asymmetric dual-path coding network model, and uses multi-level infrared image features as supervision, and fuses them with the corresponding level visible light image features in the feature fusion module of weight adaptive learning; in addition, the method also uses a high-level feature activation module to eliminate noise interference, enhance modal difference features, and combines a multi-level decoding module to obtain semantic segmentation results with clear edges and high precision; by training on a self-constructed and manually calibrated infrared and visible light power accessories dataset, a semantic segmentation model with good adaptability and strong generalization ability can be obtained, which has high engineering value. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a system module structure diagram of a method for semantic segmentation of power accessories of power transmission lines using coordinated infrared and visible light images according to the present invention;

[0066] Figure 2 A schematic diagram of the operation of a semantic segmentation method for power accessories of power transmission lines using the collaboration of infrared and visible light images.

[0067] Figure 3 Schematic diagram of the edge strength detection module, where (a) is the image erosion operation and (b) is the image dilation operation;

[0068] Figure 4 This is a schematic diagram of the semantic segmentation module for power accessories;

[0069] Figure 5 Schematic diagram of residual network block;

[0070] Figure 6 Schematic diagram of the feature fusion module for weight adaptive learning;

[0071] Figure 7 Schematic diagram of the high-level feature activation module. DETAILED DESCRIPTION

[0072] The following describes in detail a specific implementation of a method for semantic segmentation of power accessories of power transmission lines using coordinated infrared and visible light images according to the present invention, with reference to the accompanying drawings.

[0073] like Figure 1 As shown, the present invention proposes a semantic segmentation method for electric accessories based on the collaboration of infrared and visible light images, which is composed of an edge strength-based image registration module and an electric accessories semantic segmentation module. The image registration module covers algorithm parts such as edge strength detection, feature selection and descriptor matching. The electric accessories semantic segmentation module mainly includes a sample set construction module, a multi-level modal information feature extraction module, a feature fusion module of weight adaptive learning, a high-level feature activation module, a multi-level decoding module and a network training module. The sample set construction module and the network training module are respectively used to construct a complete infrared and visible light image data set pair and for training the network. Through the combination of modules, an asymmetric dual-path coding network model is constructed to extract stable image semantic information from infrared images and integrate it into visible light image features to achieve all-weather and efficient semantic segmentation of electric accessories.

[0074] like Figure 2 As shown, the specific operation process of the power accessories semantic segmentation method based on the coordination of infrared and visible light images of the present invention is as follows:

[0075] Step (1): Use a drone equipped with an infrared thermal imager and a visible light camera to circle the power tower and capture infrared and visible light image pairs of power accessories to obtain an infrared and visible light original sample set containing horizontal insulators, vertical insulators, insulation clamps, shock absorbers, and tower targets;

[0076] Step (2): The infrared and visible light original sample set captured in step (1) is processed using an edge intensity-based image registration module. The edge intensity-based image registration module includes an edge intensity detection module, a feature selection module, and a descriptor matching module. The specific process is as follows:

[0077] The edge strength detection module contains the following steps:

[0078] (1) Figure 3 As shown in the figure, firstly, the edge features of the infrared image are extracted based on the morphological gradient. The infrared image A is eroded by the structuring element B as follows:

[0079]

[0080] Each element in B has a value of 0 or 1 and can form an image of any shape. Here, B is a 3×3 convolution kernel; B z is the translation of vector z with respect to B, specifically E is the image after corrosion; similarly, the infrared image A is expanded by the structuring element B as follows:

[0081]

[0082] Among them B s ={x∈F|-x∈B}, where F is the expanded image. The process can be regarded as rotating the structuring element B 180 degrees around the origin and then corroding the infrared image A.

[0083] (2) According to (1), the inner edge D1 = A-(A!B) and the outer edge of the infrared image can be obtained Then obtain the basic gradient edge of the infrared image

[0084]

[0085] The feature selection module performs feature selection based on the ORB (Oriented FAST and rotated BRIEF) algorithm. The specific process is as follows:

[0086] (1) Take a pixel block p of size S×S and obtain the key point q=(x,y) on the pixel block through smoothing. T The gray value p(q) at , and define the binary judgment standard Γ:

[0087]

[0088] where r = (x′, y′) T is the selected threshold key point, p(r) is the grayscale value at r;

[0089] (2) Select n pairs of position coordinates (xi ,y i ), and compare it with the binary judgment standard Γ in (1) to obtain the n-dimensional feature descriptor f n (p):

[0090]

[0091] (3) A 2×n-order matrix M is transformed into a rotation matrix R θ Perform the operation to obtain the new matrix M θ for:

[0092]

[0093] in The rotation matrix R corresponding to the main direction θ of the key point q θ Expressed as

[0094]

[0095] (4) From (3), the matrix M becomes a directed form, that is, the feature descriptor has rotation invariance, and the binary descriptor is:

[0096] g n (p,θ)=f n (p)|(x i ,y i )∈S θ (8);

[0097] The descriptor matching module performs descriptor matching to achieve infrared and visible light image registration. The specific process is as follows:

[0098] (1) Using the Hamming distance as the similarity measure of two points, the brute force matching method is used to roughly match the feature points;

[0099] (2) Randomly select a pair of matching points and select a fixed 4×4 neighborhood based on the image position information. The binary mutual information I(A,B) is set to 0.4 to filter out the wrong feature points.

[0100] I(A,B)=H(A)+H(B)-H(A,B) (9),

[0101] Represents the mutual information between image A and image B, H(A) and H(B) represent the information entropy of image A and image B respectively. Taking image A as an example, p i It represents the ratio of the image with gray value i in image A, and the same applies to H(B);

[0102]

[0103] represents the joint entropy of image A and image B, x, y are the grayscale values ​​of image A and image B respectively, and p(x, y) is the joint probability distribution of the grayscale of image A and image B;

[0104] (3) Solve the transformation matrix by the progressive consistent sampling method. The steps are as follows:

[0105] a. Initialize the parameters, where v is the threshold for the number of inliers, T is the threshold for the inlier error, and N is the maximum number of iterations of the algorithm. The initial values ​​of v, T, and N are set to 4, 0.02, and 2000, respectively.

[0106] b. According to the ratio of the shortest Euclidean distance to the second shortest Euclidean distance between the matching points, the quality factor α = r of the feature point is obtained min1 / r min2 , the quality factor of each pair of matching points β = 1 / αr min1 , where the shortest Euclidean distance is r min1 , the second shortest Euclidean distance is r min2 ;

[0107] c. Arrange the matching points from large to small according to the value of β and store them in D inner Set, take the first M pairs of feature points as the preset internal point set, and then randomly select four pairs from them for the calculation of the initial homography matrix H;

[0108] d. Combine the transformation matrix to test other matching point pairs and calculate the projection error E that satisfies the model. Where (x,y) T is the three-dimensional space coordinate of the feature point, (u, v) T is the coordinate of the feature point on the normalized plane. If other matching point pairs satisfy E<T, they are classified as internal points and their number k is recorded;

[0109] e. If k < v, repeat steps a. to d.; otherwise, store the set that meets the conditions in the optimal set. If the number of iterations exceeds N, obtain the set C and transformation matrix H with the largest number of inliers in the optimal consistency set.

[0110] (4) Select a matching point pair g from the internal point set C, and then randomly select another 4 pairs of matching points. Use the transformation matrix H to calculate the transformed positions of the five matching point pairs in the registration images A and B and the distances a1, a2, a3, a4 and b1, b2, b3, b4 from the other 4 pairs of matching points to g. If b i -a i If ε is less than ε, the matching point pair is retained, otherwise it is removed, where ε is a pre-set threshold and i = 1, 2, 3, 4;

[0111] (5) Use the RANSAC algorithm to calculate the final transformation matrix H in the high-precision interior point set, and transform the image according to the transformation matrix H to find the common area between the two images;

[0112] (6) Affine transformation is used as the solution model for registration, and its mathematical expression is:

[0113]

[0114] Where (x1, y1) represents the original coordinates, (x2, y2) represents the coordinates after projection transformation, H1 represents the affine transformation matrix, (a 02 ,a 12 ) represents the offset;

[0115] Step (3): Use Figure 4 The power accessories semantic segmentation module shown in the figure trains the sample set formed by the registered and annotated infrared and visible light images to achieve the semantic segmentation task of the infrared and visible light images. The specific steps are as follows:

[0116] The registered infrared and visible light image pairs were fed into the sample set construction module to construct an infrared and visible light power accessories sample set. The LabelMe annotation tool was used to manually calibrate all visible light and corresponding infrared images, with five target categories as horizontal insulators, vertical insulators, insulation clamps, shock absorbers, and towers. The remaining unlabeled areas were used as background, resulting in an infrared and visible light image power accessories sample set.

[0117] Step (4): including the sample set construction module, the multi-level modal information feature extraction module, the weight adaptive learning feature fusion module, the high-level feature activation module, the multi-level decoding module and the network training module;

[0118] The annotated infrared and visible light images are input into the feature fusion module of multi-level modal information feature extraction and weight adaptive learning to fully extract the specific infrared and visible light data features and fuse them fully by leveraging their complementarity. The fusion results are then sent to the high-level feature activation module to enhance the differential features and eliminate the influence of interference noise. Finally, the global feature map is sent to the multi-level decoding module to obtain the final semantic segmentation result. The specific steps are as follows:

[0119] (1) The annotated infrared image and visible light image pairs are input into two parallel encoders of corresponding modalities respectively. The two parallel encoders use ResNet50 as the backbone network, and the lower branch encoder changes the number of input channels of the first convolution layer to 1 to match the corresponding infrared grayscale image. The upper branch is used to extract the corresponding visible light image features;

[0120] (2) From (1), we can obtain the corresponding feature maps of 5 different levels, namely R i With T i , where i = 1, 2, 3, 4, 5;

[0121] (3) Starting from the second layer, the infrared image is sent to the weight adaptive learning feature fusion module located before each convolution layer of the visible light encoder, and the fusion result is used as the input of the next convolution layer of the visible light encoder to achieve the auxiliary supervision role of the infrared image on the visible light image, such as Figure 5 、 6 As shown, the specific fusion steps are as follows:

[0122] a. Let the input visible light image feature be R i-1 , the infrared image feature is T i-1 , R i-1 With T i-1 After the feature connection, it is sent to a 3×3 convolution layer to reduce the number of channels to 1 / 4 of the original, denoted as P i ;

[0123] b. Use four maximum pooling layers of size 1, 5, 9, and 13 to process P i , we get four feature maps of different types and target sizes, and then send them to two convolutional layers, each containing a 3×3 convolution kernel and a ReLU activation function, to get four feature maps v j ,j=1,2,3,4;

[0124] c. Upsample the result of b. until it is equal to P i Keep consistent to form the residual structure, and P i After the number of channels is reduced to 2 and the up-sampled v j Perform pixel-level summation operation to fuse and obtain the fusion image U;

[0125] d. Input the result in c. into two 3×3 convolutional layers and activate the softmax function to obtain a dual-channel mapping W i ;

[0126] e. Separate the result in d. into two weight maps W i RGB and W i T And respectively with the input R i-1 With T i-1 Perform pixel-level multiplication to obtain the input of the next visible light encoder convolution layer

[0127] f. At each intermediate stage of the feature extraction encoder, a feature fusion module with adaptive weight learning is used to integrate infrared image features into visible light image features. The result is then fed into the next multi-level modal information feature extraction module for feature extraction. Steps a. to e. are iterated continuously until the final fusion result R5 is obtained.

[0128] (4) Then the final fusion result R5 in (3) is sent to the high-level feature activation module to enhance the difference features and eliminate the influence of interference noise, such as Figure 7 As shown, the specific steps are as follows:

[0129] a. Split R5 into two independent feature descriptors M avg With M max , and respectively M avg With M max It is fed into the adaptive average pooling layer and the adaptive maximum pooling layer to obtain an output result of size 1×1, where

[0130] M avg =AvgPool1(R5),M max =MaxPool1(R5) (12);

[0131] b. Send the two descriptors in a to the shared block, which is implemented by two convolutional layers and a ReLU function. The size of both convolutional layers is 1×1. The first convolutional layer reduces the number of channels to half of the original number. After the ReLU function, the second convolutional layer restores it. Then, the channel attention feature map is generated by element summation, sigmoid function and element multiplication:

[0132]

[0133] c. The result of b. is F. c The feature is fed into the Atrous Spatial Pyramid Pooling (ASPP) module to expand the feature receptive field and capture multi-scale information. The specific steps are as follows:

[0134] c1. F c The 3×3 convolution layers with strides of 1, 5, 9, and 13 are fed into the layers respectively, and the corresponding four multi-scale regional features V are generated. i , i=1,2,3,4;

[0135] c2. F c Send it to an adaptive average pooling layer and two 3×3 convolution layers, and upsample the result using bilinear interpolation to obtain the adaptive regional feature V5;

[0136] c3. Concatenate the feature maps of c1. and c2. to obtain the adaptive global feature map G, and use a 1×1 convolutional layer to derive its channel number;

[0137] (5) Finally, the global feature map G is sent to the multi-level decoding module to obtain the final semantic segmentation result. The specific steps are as follows:

[0138] a. Taking the first decoder as an example, the global feature map G, the corresponding visible light image encoder output feature C, and the output O of the previous decoding module are fed into the decoder. Since the first decoder does not have the output of the previous decoding module, the global feature map G is used instead;

[0139] b. In this example, C is R4, and in the other modules, they are R3 and R2 respectively. The output O of the previous decoding module and the output feature C of the visible light image encoder are fed into the adaptive spatial channel attention module to enhance its modality-specific features;

[0140] c. Upsample the result in b. to the same resolution as feature C, and reduce the number of channels of the three feature maps to 1 / 4 of the number of channels of feature map O;

[0141] d. Perform pixel-wise summation on the three results from c. and reconstruct the summed feature map using a 3×3 convolution kernel.

[0142] e. Repeat the above steps until the last decoder, which is fed into a 1×1 convolution kernel and then upsampled to change its number of channels to the number of segmentation target categories, to obtain the final semantic segmentation result Y;

[0143] Step (5): For the infrared and visible light power accessories dataset obtained in step (4), the annotated infrared and visible light image groups are randomly divided into training sets and test sets in a ratio of 4:1, where the sizes of the infrared and visible light images are both 640×480. The above network is trained using the training set, and the prediction results Y = {Y(i), i = 1, 2, ..., m} and the true value image Y are given. label ={Y label (i), i=1,2,...,m}, and use the cross entropy loss function as supervision to update the network parameters to obtain better segmentation effect:

[0144]

[0145] The optimizer uses the Adam optimization strategy, the decay parameter value is set to 0.99, the initial learning rate is 0.001, and the training iterations are 8000.

[0146] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit the present invention. Those skilled in the art will appreciate that they may modify or substitute equivalents for the specific embodiments of the present invention, provided that such modifications or variations are within the scope of protection of the pending claims.

Claims

1. A method for semantic segmentation of power accessories of transmission lines, characterized by: The method for semantic segmentation of power accessories of transmission lines comprises the following steps: Step (1): Use a drone equipped with an infrared thermal imager and a visible light camera to circle the power tower and capture infrared and visible light image pairs of power accessories to obtain an infrared and visible light original sample set containing horizontal insulators, vertical insulators, insulation clamps, shock absorbers, and tower targets; Step (2): using an edge intensity-based image registration module to process the infrared and visible light original sample set to generate an infrared and visible light image registration result with a high matching degree; Step (3): Send the registered infrared and visible light image pairs to the sample set construction module to construct the infrared and visible light power accessories sample set. Use the labelme annotation tool to manually calibrate all visible light and corresponding infrared images for the five types of targets in step (1). The remaining unlabeled areas are used as the background to obtain the infrared and visible light image power accessories sample set. Step (4): The annotated infrared and visible light image pairs are input into the feature fusion module of multi-level modal information feature extraction and weight adaptive learning to fully extract the specific infrared and visible light data features and fully fuse them by utilizing their complementarity. The fusion results are then sent to the high-level feature activation module to enhance the differential features and eliminate the influence of interference noise. Finally, the global feature map is sent to the multi-level decoding module to obtain the final semantic segmentation result. Specifically, it includes the sample set construction module, the multi-level modal information feature extraction module, the feature fusion module of weight adaptive learning, the high-level feature activation module, the multi-level decoding module and the network training module, and comprises the following steps: (1) The annotated infrared image and visible light image pairs are input into two parallel encoders of corresponding modalities respectively. The two parallel encoders use ResNet50 as the backbone network, and the lower branch encoder changes the number of input channels of the first convolution layer to 1 to match the corresponding infrared grayscale image. The upper branch is used to extract the corresponding visible light image features; (2) Starting from the second layer, the infrared image is fed into the weight adaptive learning feature fusion module located before each convolutional layer of the visible light encoder, and the fusion result is used as the input of the next convolutional layer of the visible light encoder to achieve auxiliary supervision of the infrared image on the visible light image; (3) sending the final fusion result of (2) to the high-level feature activation module to enhance the difference features and eliminate the influence of interference noise, and obtain a global feature map; (4) sending the global feature map to the multi-level decoding module to obtain the final semantic segmentation result; The feature fusion module of the weight adaptive learning includes the following steps: (1) Let the input visible light image feature be R i-1 , the infrared image feature is T i-1 , R i-1 With T i-1 After the feature connection, it is sent to a 3×3 convolution layer to reduce the number of channels to 1 / 4 of the original, denoted as P i ; (2) Use four maximum pooling layers with sizes of 1, 5, 9, and 13 to process P i , we get four feature maps of different types and target sizes, and then send them to two convolutional layers, each containing a 3×3 convolution kernel and a ReLU activation function, to get four feature maps v j ,j=1,2,3,4; (3) Upsample the result of (2) until it matches P i Keep consistent to form the residual structure, and P i After the number of channels is reduced to 2 and the up-sampled v j Perform pixel-level summation operation to fuse and obtain the fusion image U; (4) Input the result in (3) into two 3×3 convolutional layers and activate the softmax function to obtain the dual-channel mapping W i ; (5) Separate the result in (4) into two weight maps W i RGB and W i T And respectively with the input R i-1 With T i-1 Perform pixel-level multiplication to obtain the input of the next visible light encoder convolution layer (6) In each intermediate stage of the feature extraction encoder, the feature fusion module of the weight adaptive learning is used to integrate the infrared image features into the visible light image features, and the result is sent to the next multi-level modal information feature extraction module for feature extraction, and steps (1) to (5) are continuously iterated until the final fusion result R5 is obtained; The high-level feature activation module includes the following steps: (1) Split R5 into two independent feature descriptors M avg With M max , and respectively M avg With M max It is fed into the adaptive average pooling layer and the adaptive maximum pooling layer to obtain an output result of size 1×1, where M avg =AvgPool1(R5),M max =MaxPool1(R5); (2) The two descriptors in (1) are sent to the shared block, which is implemented by two convolutional layers and a ReLU function. The size of the two convolutional layers is 1×1. The first convolutional layer reduces the number of channels to half of the original number. After the ReLU function, the second convolutional layer restores it. Then, the channel attention feature map is generated by element summation, sigmoid function and element multiplication: (3) The result of (2) F c The result is fed into the Atrous Spatial Pyramid Pooling (ASPP) module to expand the feature receptive field and capture multi-scale information. The up-sampled results are then concatenated to obtain an adaptive global feature map G, and a 1×1 convolutional layer is used to derive its channel number. Step (5): The infrared and visible light image sample sets in step (3) are randomly divided into a training set and a test set in a ratio of 4:1, and the semantic segmentation network is trained using the training set. The network parameters are updated with the cross entropy loss function as supervision to obtain a more accurate segmentation effect.

2. A method for semantic segmentation of power accessories of a transmission line according to claim 1, characterized in that: In step (2), the edge strength-based image registration module includes an edge strength detection module, a feature selection module, and a descriptor matching module.

3. A method for semantic segmentation of power accessories of a transmission line according to claim 2, characterized in that: The edge strength detection module comprises the following steps: (1) The original infrared and visible light image pair is input to extract the edge features of the infrared image. The infrared image A is eroded by the structuring element B as follows: Among them B z is the translation of vector z with respect to B, specifically E is the image after corrosion; similarly, the infrared image A is expanded by the structuring element B as follows: Among them B s ={x∈F|-x∈B}, where F is the expanded image. The process can be regarded as rotating the structuring element B 180 degrees around the origin and then corroding the infrared image A. (2) According to (1), the inner edge D1 = A-(A!B) and the outer edge of the infrared image can be obtained Then obtain the basic gradient edge of the infrared image 4. The method for semantic segmentation of power accessories of a transmission line according to claim 2, characterized in that: The feature selection module performs feature selection based on the ORB algorithm and includes the following steps: (1) Take a pixel block p of size S×S and obtain the key point q=(x,y) on the pixel block through smoothing. T The gray value p(q) at , and define the binary judgment standard Γ: where r = (x′, y′) T is the selected threshold key point, p(r) is the grayscale value at r; (2) Select n pairs of position coordinates (x i ,y i ), and compare them according to the binary judgment standard Γ to obtain the n-dimensional feature descriptor f n (p): (3) A 2×n-order matrix M is transformed into a rotation matrix R θ Perform the operation to obtain the new matrix M θ for: in The rotation matrix R corresponding to the main direction θ of the key point q θ Expressed as (4) From (3), the matrix M becomes a directed form, that is, the feature descriptor has rotation invariance, and the binary descriptor is: g n (p,θ)=f n (p)|(x i ,y i )∈S θ ; Where p represents an S×S pixel block, θ represents the main direction angle of the pixel block at the key point q, and f n (p) represents the n-dimensional binary feature descriptor, (x i ,y i ) represents the coordinates of the i-th pair of sampling points selected from the neighborhood of the key point q, Represents the original sampling point set (x i ,y i ) is the new point set obtained after the rotation angle is θ.

5. The method for semantic segmentation of power accessories of a transmission line according to claim 2, characterized in that: The descriptor matching module comprises the following steps: (1) Using the Hamming distance as the similarity measure of two points, the brute force matching method is used to roughly match the feature points; (2) Solve the transformation matrix by the progressively consistent sampling method; (3) Using the RANSAC algorithm to calculate the final transformation matrix H in the high-precision interior point set, and transforming the image according to the transformation matrix H to find the common area between the two images; (4) Affine transformation is used as the solution model for registration, and its mathematical expression is: Where (x1, y1) represents the original coordinates, (x2, y2) represents the coordinates after projection transformation, represents the affine transformation matrix, (a 02 ,a 12 ) represents the offset, (a 00 ,a 11 ) represents the scaling ratio, (a 01 ,a 10 ) represents the parameters related to rotation and shear transformation.

6. The method for semantic segmentation of power accessories of a transmission line according to claim 1, characterized in that: The multi-level decoding module comprises the following steps: (1) Taking the first decoder as an example, the global feature map G, the corresponding visible light image encoder output feature C and the output O of the previous decoding module are fed into the decoder. Since the first decoder does not have the output of the previous decoding module, the global feature map G is used instead; (2) In this example, C is R4, and in the other modules, they are R3 and R2 respectively. The output O of the previous decoding module and the output feature C of the visible light image encoder are fed into the adaptive spatial channel attention module to enhance its modality-specific features; (3) Upsample the result in (2) to a resolution consistent with feature C, and reduce the number of channels of the three feature maps to 1 / 4 of the number of channels of feature map O; (4) Perform pixel-level summation on the three results in (3) and reconstruct the summed feature map using a 3×3 convolution kernel; (5) Repeat the above steps until the last decoder, which is fed into a 1×1 convolution kernel and then upsampled to change its number of channels to the number of segmentation target categories, and the final semantic segmentation result Y is obtained.

7. The method for semantic segmentation of power accessories of a transmission line according to claim 1, characterized in that: In step (5), the annotated infrared and visible light image groups are randomly divided into the training set and the test set in a ratio of 4:1, wherein the sizes of the infrared and visible light images are both 640×480. The semantic segmentation network is trained using the training set. Given the prediction result Y = {Y(i), i = 1, 2, ..., m} and the true value map Y label ={Y label (i), i=1,2,...,m}, and use the cross entropy loss function as supervision to update the network parameters to obtain better segmentation effect.

Images