Infrared and visible image fusion method for power transmission line based on feature information fusion

Abstract

The application discloses a kind of based on feature information fusion transmission line infrared and visible light image fusion method, this method is based on the effective fusion of improved convolution network realization power transmission line infrared and visible light image, the specific improvement of this improved convolution network includes: ①feature information enhancement fusion module is added between the encoding unit of the two branches of encoder, for pre-fusion before the feature map fusion of two branches, while enhancing infrared and visible light feature map, so that the fused image has more abundant and comprehensive data;②the original fusion module is improved into multi-scale feature information fusion module, the module uses different pooling layer to obtain feature context information, for obtaining higher quality fused image.The application can extract and pre-fuse the feature information of infrared and visible light image, and use different pooling layer to capture multi-scale deep features to enhance the fusion effect, obtain higher quality fused image.

Description

Technical Field

[0001] This invention relates to the technical field of information fusion, and in particular to a method for fusing infrared and visible light images of power transmission lines based on feature information fusion. Background Technology

[0002] Power transmission lines are a crucial component of the power system, and their stable operation is essential for maintaining the safety and reliability of the power grid. Traditionally, grid operators assess the condition of transmission lines by monitoring infrared or visible light images. Infrared images reflect the temperature distribution and thermal changes of the lines, while visible light images provide information on the line structure and surface details.

[0003] However, using infrared or visible light images alone has certain limitations, often failing to provide a comprehensive description of the entire scene. Infrared images are advantageous for detecting anomalies such as high temperatures, corrosion, and insufficient light, but their ability to identify certain structural problems or object details is relatively weak. While visible light images can provide rich structural information, they may be limited at night or in severe weather conditions and cannot directly reflect temperature changes in the transmission line. Currently, several image fusion methods exist, attempting to combine infrared and visible light images to compensate for their respective shortcomings. Infrared and visible light image fusion aims to utilize the different imaging characteristics of infrared and visible light images to mine complementary information between multimodal data, synthesizing a new image that can completely and accurately describe the actual scene. Effective fusion of these two types of images can contain richer information, improving the efficiency of subsequent technologies such as transmission line target recognition, image registration, visual tracking, and 3D reconstruction.

[0004] However, existing fusion methods still have some shortcomings. Some methods simply superimpose the two images, failing to fully utilize their feature information, resulting in low quality and resolution of the fused image. Other traditional methods use related mathematical transformations and manually designed fusion rules to obtain the fused image; these methods are often only applicable to specific fusion tasks and lack adaptability to complex situations. Currently, deep learning-based methods have become the mainstream approach for image fusion. Due to the strong fitting ability of neural networks, these methods can usually achieve better fusion results by designing objective functions to extract and reconstruct effective information. Although existing deep learning-based methods have achieved better fusion performance compared to traditional methods, they often neglect the critical path features of the infrared and visible light images themselves during processing, leading to the introduction of a lot of invalid information during the fusion process, and there is still room for further improvement in the quality of the fused image. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for fusing infrared and visible light images of transmission lines based on feature information fusion. This method can enhance the fusion effect by extracting and pre-fusing feature information from infrared and visible light images and using different pooling layers to capture multi-scale deep features, thereby obtaining a higher quality fused image.

[0006] To achieve the above objectives, the technical solution provided by this invention is: a method for fusing infrared and visible light images of power transmission lines based on feature information fusion. This method is based on an improved convolutional network to achieve effective fusion of infrared and visible light images of power transmission lines. The specific improvements of the improved convolutional network include: ① adding a feature information enhancement fusion module between the encoding units of the two branches of the encoder, which is used to pre-fuse the feature maps of the two branches before fusion, and simultaneously enhance the infrared and visible light feature maps, so that the fused image has richer and more comprehensive data; ② improving the original fusion module to a multi-scale feature information fusion module, which uses different pooling layers to obtain feature context information, in order to obtain a higher quality fused image.

[0007] The specific implementation of the infrared and visible light image fusion method for transmission lines includes:

[0008] The acquired infrared and visible light images of the power grid transmission lines are input into the trained improved convolutional network for the following operations:

[0009] Infrared and visible light images are input into an encoder, where two encoding units process them separately. Specifically, one encoding unit processes the infrared image, and the other processes the visible light image. Each encoding unit consists of one convolutional layer and three residual convolutional modules. Two feature enhancement and fusion modules are added between the two encoding units. First, the input infrared and visible light images are pre-convolved using the convolutional layers of the two encoding units. Then, they are input into their respective first residual convolutional modules for feature extraction, obtaining first infrared and visible light feature maps. These first infrared and visible light feature maps are then input into the first feature enhancement and fusion module, which outputs enhanced feature representations of the first infrared and visible light, respectively. These enhanced feature representations are then concatenated with their respective first infrared and visible light feature maps to generate enhanced first infrared and visible light feature maps. The visible light feature enhancement maps are input into their respective second residual convolutional modules for feature extraction, resulting in second infrared and visible light feature maps. These second infrared and visible light feature maps are then input into a second feature information enhancement and fusion module, which outputs enhanced feature representations of the second infrared and visible light respectively. These enhanced feature representations are then concatenated with their respective second infrared and visible light feature maps to generate enhanced second infrared and visible light feature enhancement maps. These enhanced second infrared and visible light feature enhancement maps are then input into their respective third residual convolutional modules for feature extraction, resulting in third infrared and visible light feature maps. The third infrared and visible light feature maps obtained after feature extraction are directly input into a multi-scale feature information fusion module. This module uses multiple max pooling layers to pool the third infrared and visible light feature maps, and then expands the network's receptive field through concatenated convolutions to complete the fusion of the third infrared and visible light feature maps, resulting in a hybrid feature map.

[0010] The hybrid feature map is input into the decoder for feature dimensionality reduction. The decoder also uses three residual convolution modules for decoding to obtain the reconstructed infrared and visible light fused image of the power grid transmission line.

[0011] Furthermore, the feature information enhancement and fusion module enhances the input infrared and visible light feature maps, as detailed below:

[0012] Because infrared images highlight target information, the Frequency-tuned algorithm is used to perform saliency processing on the infrared feature map, outputting an infrared saliency feature map. Because visible light images have rich texture information, the Sobel operator is used to extract the gradient map from the visible light feature map, outputting a visible light gradient feature map. The infrared feature map, visible light feature map, infrared saliency feature map, and visible light gradient feature map are concatenated. The concatenated feature map is input into two branches. Each branch first adjusts the number of channels through a 1×1 convolution, and then processes it through a channel attention mechanism to output enhanced infrared and visible light feature representations.

[0013] The processing procedure of the feature information enhancement module is as follows:

[0014] F m =concat(concat(F I Sobel(F) I )),concat(F V ,FT(F V )))

[0015] F I '=SE(conv(F m ))

[0016] F V '=SE(conv(F m ))

[0017] In the formula, F I F V F represents the input infrared and visible light feature maps, respectively. m F represents the concatenated feature map. I '、F V 'represents the infrared and visible light enhanced feature representations output by the two branches after processing by the channel attention mechanism, respectively; Sobel represents the gradient map of the image extracted using the Sobel operator; FT represents the saliency map of the image extracted using the Frequency-tuned algorithm; and SE represents the enhanced feature map using the SE channel attention mechanism.

[0018] Furthermore, the multi-scale feature information fusion module first performs max pooling on the input infrared and visible light feature maps through two branches, with each branch having two 3×3 max pooling layers connected in series. Multi-scale feature information fusion is achieved by splicing the intermediate feature maps during the pooling process. Finally, the feature maps from the two branches are spliced ​​and fused together to obtain a hybrid feature map.

[0019] The processing procedure of the multi-scale feature information fusion module is as follows:

[0020]

[0021] F m '=conv(concat(F mI ,F mV ))

[0022] In the formula, These represent the infrared and visible light feature maps input to the multi-scale feature information fusion module, respectively. MaxPool 3×3 This indicates that the feature map is subjected to 3×3 max pooling, F mI F mV F represents the infrared and visible light feature maps after pooling and stitching through multiple max pooling layers, respectively. m 'This represents the hybrid feature map output by the multi-scale feature information fusion module.

[0023] Furthermore, the improved convolutional network utilizes the pixel loss and gradient loss between the reconstructed infrared and visible light fused image and the original infrared and visible light images to construct a total loss function. The pixel loss is used to preserve information from the original image, and the gradient loss is used to preserve the maximum texture detail of the original image. The formula for this total loss function is as follows:

[0024] L = L pixel +λL grad

[0025]

[0026] In the formula, L represents the total loss function, L pixel L grad I represents pixel loss and gradient loss, respectively. I I V Representing the original infrared and visible light images respectively, I F This represents the reconstructed infrared and visible light fused image, where H and W represent the image height and width, respectively. and These represent the gradient maps of the original infrared image, the original visible light image, and the reconstructed fused infrared and visible light image, respectively, where λ is a hyperparameter.

[0027] Furthermore, the residual convolution module consists of three convolutional layers and an equivalent mapping of a skip connection.

[0028] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0029] 1. Enhanced Information Richness. This invention pre-fuses infrared and visible light feature images through a feature information enhancement fusion module to obtain richer and more comprehensive data. Infrared images provide thermal information such as temperature, while visible light images provide richer structural and detailed information. The fused image can simultaneously reflect temperature distribution and object structure, thus enhancing the information richness of transmission lines.

[0030] 2. Retaining Important Information. The feature information enhancement and fusion module adds a channel attention mechanism after pre-fusion, which can adaptively enhance and retain information that plays a key role in the assessment of transmission line status according to the characteristics and needs of the image, avoiding unnecessary interference and noise.

[0031] 3. Optimize image quality. In addition to using a feature information enhancement fusion module to improve image fusion quality, this invention uses a multi-scale feature information fusion module, which can effectively integrate feature context information, providing a more comprehensive and richer visual context for subsequent image decoding, thereby improving the quality of the fused image. Attached Figure Description

[0032] Figure 1 The diagram shows the structure of an improved convolutional network; in the diagram, ENCODER represents the encoder, DECODER represents the decoder, and I... I and I V These represent the input infrared and visible light images, respectively. 5×5Conv represents a convolutional layer with a 5×5 kernel. ResBlock represents the residual convolution module. Fusion1 represents the feature enhancement and fusion module. Fusion2 represents the multi-scale feature fusion module. F 'c' represents the reconstructed infrared and visible light fused image, and 'c' represents the feature map stitching.

[0033] Figure 2 The diagram shows the structure of the residual convolution module; in the diagram, 1×1Conv and 3×3Conv represent convolutional layers with kernels of 1×1 and 3×3, respectively.

[0034] Figure 3 This is a structural diagram of the feature information enhancement and fusion module; in the diagram, F I and F Vrepresents the infrared and visible light feature maps of the input module, respectively. Sobel Operator gradient extraction represents the gradient map extracted using the Sobel operator. Frequency-tuned Salient Detection represents the saliency processing of the feature map using the Frequency-tuned algorithm. Channel Attention represents the enhancement of the feature map using the SE channel attention mechanism. 1×1Conv represents a convolutional layer with a 1×1 kernel. c represents feature map concatenation.

[0035] Figure 4 This is a structural diagram of the multi-scale feature information fusion module; in the diagram, F I and F V represents the infrared and visible light feature maps of the input module, respectively; 3×3Maxpool represents the max pooling layer with a 3×3 kernel; 1×1Conv represents the convolutional layer with a 1×1 kernel; and c represents the feature map concatenation. Detailed Implementation

[0036] The present invention will be further described below with reference to specific embodiments.

[0037] This embodiment discloses a method for fusing infrared and visible light images of power transmission lines based on feature information fusion. This method utilizes an improved convolutional network to effectively fuse infrared and visible light images of power grid transmission lines. Figure 1 As shown, the network consists of two parts: an encoder and a decoder. Specific improvements include: ① Adding a feature information enhancement and fusion module (Fusion1) between the coding units of the two branches of the encoder. This module pre-fuses the feature maps of the two branches before fusion, simultaneously enhancing both infrared and visible light feature maps, resulting in a richer and more comprehensive fused image; ② Improving the original fusion module to a multi-scale feature information fusion module (Fusion2). This module uses different pooling layers to obtain feature context information, leading to a higher-quality fused image. The specific implementation of this method includes:

[0038] Acquiring infrared and visible light images of power grid transmission lines I I I V The following operations are performed using the trained improved convolutional network:

[0039] a. Infrared and visible light images I I I VThe images are input separately to the encoder, where two encoding units process the infrared and visible light images respectively. Specifically, the infrared image is processed by one encoding unit, and the visible light image by the other. Each encoding unit consists of a 5×5 convolutional layer (Conv) and three residual convolutional modules (ResBlock). Two feature enhancement and fusion modules (Fusion1) are added between the two encoding units. Figure 2 As shown, each residual convolutional module (ResBlock) consists of three convolutional layers and an equivalent mapping of a skip connection. First, two 5×5 convolutional layers with coded units are used to perform preliminary convolution on the input infrared and visible light images, respectively. These images are then fed into their respective first residual convolutional modules for feature extraction, obtaining first infrared and visible light feature maps. These first infrared and visible light feature maps are then fed into a first feature information enhancement and fusion module, which outputs enhanced feature representations of the first infrared and visible light, respectively. These enhanced feature representations are then concatenated with their respective first infrared and visible light feature maps to generate enhanced first infrared and visible light feature maps. These enhanced first infrared and visible light feature maps are then fed into their respective second residual convolutional modules for feature extraction, obtaining second infrared and visible light feature maps. The second infrared and visible light feature maps are input into the second feature information enhancement and fusion module, which outputs enhanced feature representations of the second infrared and visible light, respectively. These enhanced feature representations are then concatenated with their respective second infrared and visible light feature maps to generate enhanced second infrared and visible light feature maps. These enhanced second infrared and visible light feature maps are then input into their respective third residual convolution modules for feature extraction to obtain third infrared and visible light feature maps. The third infrared and visible light feature maps obtained after feature extraction are directly input into the multi-scale feature information fusion module (Fusion2). This module uses multiple max pooling layers to pool the third infrared and visible light feature maps, and then expands the network's receptive field through convolutional convolution to complete the fusion of the third infrared and visible light feature maps, resulting in a hybrid feature map.

[0040] b. Input the hybrid feature map into the decoder for feature dimensionality reduction. The decoder also uses three residual convolutional modules (ResBlock) for decoding to obtain the reconstructed infrared and visible light fused image I of the power grid transmission line. F .

[0041] Specifically, such as Figure 3 As shown, the specific details of the feature information enhancement and fusion module (Fusion1) are as follows:

[0042] Because infrared images highlight target information, a frequency-tuned algorithm is used to saliency-enhance the feature map, outputting an infrared saliency feature map. This algorithm improves image clarity and contrast by enhancing or suppressing specific frequency components of the image signal. Since visible light images are rich in texture information, the Sobel operator is used to extract their gradient maps. By detecting edges and gradient information in the image, image features with important shapes and structures are extracted, outputting a visible light gradient feature map. The infrared feature map, visible light feature map, infrared saliency feature map, and visible light gradient feature map are concatenated. The resulting feature map is input into two branches. Each branch first adjusts the number of channels through a 1×1 convolution, and then processes the data through a channel attention mechanism to output enhanced infrared and visible light feature representations.

[0043] The processing procedure of the feature information enhancement module is represented as follows:

[0044] F m =concat(concat(F I Sobel(F) I )),concat(F V ,FT(F V )))

[0045] F I '=SE(conv(F m ))

[0046] F V '=SE(conv(F m ))

[0047] In the formula, F I F V F represents the input infrared and visible light feature maps, respectively. m F represents the concatenated feature map. I '、F V 'represents the infrared and visible light enhanced feature representations output by the two branches after processing by the channel attention mechanism, respectively; Sobel represents the gradient map of the image extracted using the Sobel operator; FT represents the saliency map of the image extracted using the Frequency-tuned algorithm; and SE represents the enhanced feature map using the SE channel attention mechanism.

[0048] Specifically, such as Figure 4As shown, the multi-scale feature information fusion module (Fusion2) first performs max pooling on the input infrared and visible light feature maps through two branches. Each branch has two 3×3 max pooling layers connected in series. Multi-scale feature information fusion is achieved by splicing the intermediate feature maps during the pooling process. Finally, the feature maps of the two branches are spliced ​​and fused through splicing convolution to obtain a hybrid feature map.

[0049] The processing procedure of the multi-scale feature information fusion module is represented as follows:

[0050]

[0051] F m '=conv(concat(F mI ,F mV ))

[0052] In the formula, These represent the infrared and visible light feature maps input to the multi-scale feature information fusion module, respectively. MaxPool 3×3 This indicates that the feature map is subjected to 3×3 max pooling, F mI F mV F represents the infrared and visible light feature maps after pooling and stitching through multiple max pooling layers, respectively. m 'This represents the hybrid feature map output by the multi-scale feature information fusion module.

[0053] Specifically, the improved convolutional network utilizes the pixel loss and gradient loss between the reconstructed infrared and visible light fused image and the original infrared and visible light images to construct the total loss function. The pixel loss is used to preserve the information of the original image, and the gradient loss is used to preserve the maximum texture details of the original image. The formula for this total loss function is as follows:

[0054] L = L pixel +λL grad

[0055]

[0056] In the formula, L represents the total loss function, L pixel L grad I represents pixel loss and gradient loss, respectively. I I V Representing the original infrared and visible light images respectively, I F This represents the reconstructed infrared and visible light fused image, where H and W represent the image height and width, respectively. and These represent the gradient maps of the original infrared image, the original visible light image, and the reconstructed fused infrared and visible light image, respectively, where λ is a hyperparameter.

[0057] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Therefore, any changes made in accordance with the shape and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for fusing infrared and visible light images of transmission lines based on feature information fusion, characterized in that, include: The acquired infrared and visible light images of the power grid transmission lines are input into the trained improved convolutional network for the following operations: Infrared and visible light images are input into an encoder, where two encoding units process them separately. Specifically, one encoding unit processes the infrared image, and the other processes the visible light image. Each encoding unit consists of one convolutional layer and three residual convolutional modules. Two feature enhancement and fusion modules are added between the two encoding units. First, the input infrared and visible light images are pre-convolved using the convolutional layers of the two encoding units. Then, they are input into their respective first residual convolutional modules for feature extraction, obtaining first infrared and visible light feature maps. These first infrared and visible light feature maps are then input into the first feature enhancement and fusion module, which outputs enhanced feature representations of the first infrared and visible light, respectively. These enhanced feature representations are then concatenated with their respective first infrared and visible light feature maps to generate enhanced first infrared and visible light feature maps. The visible light feature enhancement maps are input into their respective second residual convolutional modules for feature extraction, resulting in second infrared and visible light feature maps. These second infrared and visible light feature maps are then input into a second feature information enhancement and fusion module, which outputs enhanced feature representations of the second infrared and visible light respectively. These enhanced feature representations are then concatenated with their respective second infrared and visible light feature maps to generate enhanced second infrared and visible light feature enhancement maps. These enhanced second infrared and visible light feature enhancement maps are then input into their respective third residual convolutional modules for feature extraction, resulting in third infrared and visible light feature maps. The third infrared and visible light feature maps obtained after feature extraction are directly input into a multi-scale feature information fusion module. This module uses multiple max pooling layers to pool the third infrared and visible light feature maps, and then expands the network's receptive field through concatenated convolutions to complete the fusion of the third infrared and visible light feature maps, resulting in a hybrid feature map. The hybrid feature map is input into the decoder for feature dimensionality reduction. The decoder also uses three residual convolution modules for decoding to obtain the reconstructed infrared and visible light fused image of the power grid transmission line. The feature information enhancement and fusion module enhances the input infrared and visible light feature maps, as follows: Because infrared images highlight target information, the Frequency-tuned algorithm is used to perform saliency processing on the infrared feature map, outputting an infrared saliency feature map. Because visible light images have rich texture information, the Sobel operator is used to extract the gradient map from the visible light feature map, outputting a visible light gradient feature map. The infrared feature map, visible light feature map, infrared saliency feature map, and visible light gradient feature map are concatenated. The concatenated feature map is input into two branches. Each branch first adjusts the number of channels through a 1×1 convolution, and then processes it through a channel attention mechanism to output enhanced infrared and visible light feature representations. The processing procedure of the feature information enhancement and fusion module is represented as follows: ； ； ； In the formula, , These represent the input infrared and visible light feature maps, respectively. This represents the spliced ​​feature map. , These represent the infrared and visible light enhancement features output after the two branches are processed by the channel attention mechanism, respectively. This indicates that the gradient map of the image was extracted using the Sobel operator. This indicates that the saliency map of the image was extracted using the Frequency-tuned algorithm. , This indicates that the SE channel attention mechanism is used to enhance the feature map.

2. The method for fusing infrared and visible light images of transmission lines based on feature information fusion according to claim 1, characterized in that, The multi-scale feature information fusion module first performs max pooling on the input infrared and visible light feature maps through two branches. Each branch is connected in series with two 3×3 max pooling layers. Multi-scale feature information fusion is achieved by splicing the intermediate feature maps during the pooling process. Finally, the feature maps of the two branches are spliced ​​and fused through splicing convolution to obtain a hybrid feature map. The processing procedure of the multi-scale feature information fusion module is as follows: ； In the formula, , These represent the infrared and visible light feature maps input to the multi-scale feature information fusion module, respectively. Indicates performing a process on the feature map. Max pooling processing, , These represent the infrared and visible light feature maps after pooling and stitching through multiple max-pooling layers, respectively. This represents the final output of the multi-scale feature information fusion module, which is a hybrid feature map.

3. The method for fusing infrared and visible light images of transmission lines based on feature information fusion according to claim 1, characterized in that, The improved convolutional network utilizes the pixel loss and gradient loss between the reconstructed infrared and visible light fused image and the original infrared and visible light images to construct a total loss function. The pixel loss is used to preserve information from the original image, and the gradient loss is used to preserve the maximum texture detail of the original image. The formula for this total loss function is as follows: ； In the formula, Represents the total loss function. , These represent pixel loss and gradient loss, respectively. , These represent the original infrared and visible light images, respectively. This represents the reconstructed infrared and visible light fused image. , These represent the height and width of the image, respectively. , and These represent the gradient maps of the original infrared image, the original visible light image, and the reconstructed fused infrared and visible light image, respectively. It is a hyperparameter.

4. The method for fusing infrared and visible light images of transmission lines based on feature information fusion according to claim 1, characterized in that, The residual convolution module consists of three convolutional layers and an equivalent mapping of a skip connection.

Images