Image fusion method for AC filter detection and storage medium

Abstract

The invention discloses an image fusion method for AC filter detection and a storage medium, and relates to the technical field of image processing. The method comprises the steps of obtaining an infrared image and a visible light image of an AC filter; inputting the infrared image and the visible light image into a pre-trained fusion model, and outputting a fusion image; the fusion model comprises an encoder and a decoder, the encoder comprises two branch networks, a mutual information extraction module and a multi-scale fusion module, the two branch networks extract features of an infrared image and a visible light image respectively, the mutual information extraction module obtains complementary information according to the extracted features and screens out irrelevant information by using a channel attention mechanism, and the multi-scale fusion module carries out multi-scale fusion on the irrelevant information. And respectively injecting the screened complementary information into the two branch networks to obtain two splicing features, performing multi-scale fusion on the two splicing features by a multi-scale fusion module to obtain fusion features, and decoding the fusion features by a decoder to obtain a fusion image. Therefore, the fusion effect of the infrared image and the visible light image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image fusion method and a storage medium for AC filter detection. Background Art

[0002] In recent years, infrared and visible light image fusion technology has received extensive attention in the fields of target detection, monitoring, remote sensing, etc. Infrared images can highlight the thermal radiation characteristics of targets in low light or harsh environments, while visible light images provide rich details and texture information. However, due to the different physical properties of the two image sources, how to effectively fuse the information of the two to improve the comprehensive expression ability of the image has become a research hotspot.

[0003] As a key device in the power system, AC filters are widely used to suppress harmonic interference in the power grid and improve the quality of power. In smart grids and power transmission monitoring, infrared and visible light image fusion technology has gradually become an important means of equipment status monitoring. In practical applications, infrared images can intuitively reflect the thermal distribution state of AC filters and are used to detect overheating caused by overload or fault, while visible light images provide structural details of the equipment (such as wiring integrity and appearance damage) to support the appearance and mechanical status assessment of the equipment. Through image fusion technology, the information of the two modes can be combined to generate a comprehensive image with more diagnostic value.

[0004] Image fusion technology is mainly divided into traditional algorithms and deep learning algorithms. Traditional algorithms are mainly based on methods such as multi-scale decomposition and sparse representation. However, these methods often rely on manually designed rules and are difficult to adapt to the nonlinear relationship of multimodal images in complex scenes, resulting in limited fusion effects. Compared with traditional image processing methods, fusion models based on deep learning can automatically learn features in images and complete image fusion tasks without complex manual design steps. However, existing deep learning methods have problems such as insufficient extraction of complementary information between infrared and visible light images and feature loss, which reduces the quality of fused images. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art to at least a certain extent. To this end, one object of the present invention is to provide an image fusion method and storage medium for AC filter detection to improve the fusion effect of infrared images and visible light images, which is helpful for subsequent AC filter detection.

[0006] In a first aspect, the present invention provides an image fusion method for AC filter detection, comprising: acquiring an infrared image and a visible light image of the AC filter; inputting the infrared image and the visible light image into a pre-trained fusion model, and outputting a fused image for AC filter detection; wherein the fusion model comprises an encoder and a decoder, the encoder comprises a first branch network, a second branch network, a mutual information extraction module and a multi-scale fusion module, the first branch network is used to extract a first feature of the infrared image, the second branch network is used to extract a second feature of the visible light image, the mutual information extraction module is used to obtain complementary information based on the first feature and the second feature, and use a channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network respectively, so as to respectively splice with the first feature and the second feature to obtain a first splicing feature and a second splicing feature, the multi-scale fusion module is used to perform multi-scale fusion on the first splicing feature and the second splicing feature to obtain a fused feature, and the decoder is used to decode the fused feature to obtain a fused image.

[0007] According to one embodiment of the present invention, the first branch network and the second branch network have the same structure, including four first convolution blocks connected in sequence, and the number of the mutual information extraction modules is three, which correspond one to one with the three subsequent first convolution blocks; wherein the mutual information extraction module is used to obtain complementary information based on the first feature and the second feature output by the corresponding first convolution blocks in the two branch networks, and use the channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network, respectively, so as to perform splicing operations with the first feature and the second feature, respectively.

[0008] According to an embodiment of the present invention, the mutual information extraction module obtains the features injected into the first branch network by the following formula: in, , represents the features containing shared information, , represents the weight, , Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the first branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

[0009] According to an embodiment of the present invention, the mutual information extraction module obtains the features injected into the second branch network by the following formula: in, , represents the features containing shared information, , represents the weight, , Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the second branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

[0010] According to one embodiment of the present invention, the multi-scale fusion module includes a third branch network, a fourth branch network, and a local feature fusion unit, a first global feature fusion unit and a second global feature fusion unit connected in sequence. The third branch network and the fourth branch network have the same structure, including a second convolution block, a third convolution block and a fourth convolution block; wherein the local feature fusion module is used to perform local feature fusion on the first splicing feature and the second splicing feature processed by the second convolution block to obtain a first fused sub-feature; the first global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the third convolution block, and the first fused sub-feature to obtain a second fused sub-feature; the second global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the fourth convolution block, and the second fused sub-feature to obtain the fused feature.

[0011] According to an embodiment of the present invention, the local feature fusion unit obtains the first fusion sub-feature by the following formula: in, and represent the first splicing feature and the second splicing feature after being processed by the second convolution block, respectively, represents the first fusion sub-feature, Indicates a connection operation. and Represent 1×1 and 3×3 convolution operations respectively.

[0012] According to an embodiment of the present invention, the first global feature fusion unit and the second global feature fusion unit have the same structure, and the second global feature fusion unit obtains the second fusion sub-feature by the following formula: in, and Respectively represent the first splicing feature and the second splicing feature after being processed by the third convolution block, represents the second fusion sub-characteristic, , represents the intermediate fusion sub-feature, represents the fusion feature.

[0013] According to one embodiment of the present invention, the second convolution block includes a 1×1 convolution layer and a LeakyReLU activation layer connected in sequence, the third convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fourth convolution block includes a 5×5 convolution layer and a LeakyReLU activation layer.

[0014] According to one embodiment of the present invention, the decoder includes four first convolution blocks and one fifth convolution block connected in sequence, the first convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fifth convolution block includes a 3×3 convolution layer and a Tanh activation layer connected in sequence.

[0015] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image fusion method for AC filter detection described in the first aspect is implemented.

[0016] The image fusion method and storage medium for AC filter detection in the embodiments of the present invention design a mutual information extraction module to improve the extraction effect of complementary information. The module combines the channel attention mechanism to fuse and screen the infrared and visible light image features, and can extract complementary information while reducing the interference of irrelevant information. At the same time, in order to reduce feature loss, a multi-scale fusion module is designed. The module can improve the final fused image effect by capturing feature information at different scales and performing layered fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a structural schematic diagram of a fusion model according to an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a mutual information extraction module according to an embodiment of the present invention; Figure 3 is a schematic structural diagram of a multi-scale feature fusion module according to an embodiment of the present invention; Figure 4 is a structural schematic diagram of a local feature fusion unit according to an embodiment of the present invention; Figure 5 is a schematic structural diagram of a second global feature fusion unit according to an embodiment of the present invention; Figure 6 It is a schematic diagram of obtaining a fused image based on an infrared image and a visible light image according to some examples of the present invention. DETAILED DESCRIPTION

[0018] Aiming at the problem of insufficient complementary information extraction and feature loss in the fusion of infrared image and visible light image of AC filter, the present invention proposes an image fusion method for AC filter detection. In this method, in order to improve the extraction effect of complementary information, a mutual information extraction module is designed. This module combines the channel attention mechanism to fuse and screen the features of infrared and visible light images, and can extract complementary information while reducing the interference of irrelevant information; at the same time, in order to reduce feature loss, a multi-scale fusion module is designed. This module can improve the final fusion image effect by capturing feature information at different scales and performing layered fusion.

[0019] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0020] The following describes an image fusion method and a storage medium for AC filter detection according to an embodiment of the present invention with reference to the accompanying drawings.

[0021] In an embodiment of the present invention, the image fusion method for AC filter detection includes the following steps S11-S12: S11, acquiring an infrared image and a visible light image of the AC filter; S12, inputting the infrared image and the visible light image into a pre-trained fusion model, and outputting a fusion image for AC filter detection.

[0022] Among them, Figure 1As shown, the fusion model includes an encoder and a decoder, the encoder includes a first branch network, a second branch network, a mutual information extraction module and a multi-scale fusion module, the first branch network is used to extract the first feature of the infrared image, the second branch network is used to extract the second feature of the visible light image, the mutual information extraction module is used to obtain complementary information according to the first feature and the second feature, and use the channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network respectively, so as to be spliced ​​with the first feature and the second feature respectively to obtain the first splicing feature and the second splicing feature, the multi-scale fusion module is used to perform multi-scale fusion on the first splicing feature and the second splicing feature to obtain a fused feature, and the decoder is used to decode the fused feature to obtain a fused image.

[0023] Specifically, see Figure 1 , the fusion model is mainly divided into two parts: the first part is the encoder, in which the encoder uses branch networks (e.g., including multiple continuous convolutions) to extract the unique features of the two images respectively. Considering that the shooting scenes of the two images are the same, in order to better mine the complementary information between the two modal images, a mutual information extraction module is designed. This module first analyzes the correlation between the two modalities to effectively extract the complementary information of the input source image, and then uses the channel attention mechanism to filter out irrelevant information interference, and finally injects these complementary information into the two branch networks to enhance the complementarity between the modalities; at the same time, a multi-scale fusion module is also designed in the encoder to improve the quality of the fused image by fusing multi-scale information in multiple layers. The second part is the decoder, which reconstructs the fusion features of the infrared image and the visible light image through the decoder (e.g., using multiple continuous convolutions) to finally obtain a fused image, which can be used for AC filter detection (such as fault detection). The specific training process of the fusion model is as follows: Step 1: Collect infrared image and visible light image data of the AC filter, and randomly divide the data into training set and test set according to the sample ratio of 7:3 (or other ratios, which can be calibrated according to needs); Step 2: Input the training set into the designed fusion model to perform model training; Step 3: After the model training is completed, input the test set and observe the fusion test results.

[0024] Among them, if the test results meet the preset conditions (such as the performance indicators meet the requirements), the trained fusion model can be used for the fusion of infrared images and visible light images, otherwise the fusion model needs to continue to be trained.

[0025] In some embodiments of the present invention, see Figure 1The first branch network and the second branch network have the same structure, including four first convolution blocks connected in sequence, and the number of mutual information extraction modules is three, corresponding to the three subsequent first convolution blocks. Among them, the mutual information extraction module is used to obtain complementary information according to the first feature and the second feature output by the corresponding first convolution blocks in the two branch networks, and use the channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network respectively, so as to perform splicing operations with the first feature and the second feature respectively.

[0026] For example, see Figure 1 The first convolutional block includes a 3×3 convolutional layer and a LeakyReLU activation layer connected in sequence. Four consecutive 3×3 convolutions and LeakyReLU activation functions are used independently on the input infrared image and visible light image to extract the unique features of the two images.

[0027] In some examples of the present invention, the mutual information extraction module obtains the features injected into the first branch network through the following formula (1): (1) in, , represents the features containing shared information, , represents the weight, , Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the first branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

[0028] In some examples of the present invention, the mutual information extraction module obtains the features injected into the second branch network by the following formula: (2) in, , represents the features containing shared information, , represents the weight, 、 Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the second branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

[0029] Specifically, since the infrared image and the visible light image capture the same scene, they contain unique information and shared information. In order to better learn the complementary information between the two images, a mutual information extraction module is designed to extract the shared information of the two images. The structure of the module is as follows: Figure 2 shown.

[0030] See also Figure 2 , and They represent the features corresponding to the input infrared image and visible light image respectively. The features are first convolved and connected, and then passed through two layers of 1×1 convolution to obtain features rich in shared information, as shown in the following formula (3).

[0031] (3) in, and Represent 1×1 and 3×3 convolution operations respectively, is the connection operation. Then, in order to reduce the interference of irrelevant information, the features containing shared information are obtained Use global average pooling and channel attention mechanism to obtain weights , as shown in formula (4).

[0032] (4) in, is the channel attention mechanism, is the global average pooling operation. In addition, see Figure 2 , but also Use 3×3 convolution to obtain deeper features and combine them with weights Multiply them together and then add them to the features after 1×1 convolution to obtain features with complementary information, as shown in equations (1) and (2).

[0033] Finally, the obtained features with complementary information are injected into the corresponding branch network.

[0034] In some embodiments of the present invention, Figure 3 As shown, the multi-scale fusion module includes a third branch network, a fourth branch network, and a local feature fusion unit, a first global feature fusion unit, and a second global feature fusion unit connected in sequence. The third branch network and the fourth branch network have the same structure, including a second convolution block, a third convolution block, and a fourth convolution block.

[0035] Among them, the local feature fusion module is used to perform local feature fusion on the first splicing feature and the second splicing feature processed by the second convolution block to obtain a first fused sub-feature; the first global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the third convolution block, and the first fused sub-feature to obtain a second fused sub-feature; the second global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the fourth convolution block, and the second fused sub-feature to obtain a fused feature.

[0036] Specifically, in order to better capture features, the present invention designs a multi-scale fusion module, the structure of which is as follows: Figure 3 As shown, Figure 3 middle and They represent the first splicing feature (corresponding to the infrared image) and the second splicing feature (corresponding to the visible light image) input into the module respectively. In this module, different convolution kernels (1×1, 3×3, 5×5) are used for the first splicing feature and the second splicing feature to capture feature information of different scales, and the activation function uses LeakyReLU (LReLU in the figure). Subsequently, these multi-scale features are input into three fusion units (i.e., local feature fusion unit, first global feature fusion unit, and second global feature fusion unit) for multi-layer feature fusion.

[0037] Exemplarily, the second convolution block includes a 1×1 convolution layer and a LeakyReLU activation layer connected in sequence, the third convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fourth convolution block includes a 5×5 convolution layer and a LeakyReLU activation layer.

[0038] In some examples of the present invention, the local feature fusion unit obtains the first fusion sub-feature by the following formula (5): (5) in, and Respectively represent the first splicing feature and the second splicing feature after being processed by the second convolution block, represents the first fusion sub-feature, Indicates a connection operation. and Represent 1×1 and 3×3 convolution operations respectively.

[0039] Specifically, the structure of the local feature fusion unit is as follows: Figure 4 As shown, Figure 4 middle and They represent the first concatenated feature and the second concatenated feature of the local feature fusion unit after 1×1 convolution input, respectively. Indicates the first fusion sub-characteristic. Figure 4 In the local feature fusion unit, the two splicing features are first connected and then processed in two paths. Convolution operation generates low-dimensional features; in the second path, and Operation is performed to achieve deeper information extraction, as shown in the above formula (5).

[0040] In some examples of the present invention, the first global feature fusion unit and the second global feature fusion unit have the same structure, and the second global feature fusion unit obtains the second fusion sub-feature by the following formula (6): (6) in, and They represent the first concatenated feature and the second concatenated feature after being processed by the third convolutional block, respectively. represents the second fusion sub-feature, , represents the intermediate fusion sub-feature, Represents fusion features.

[0041] Specifically, considering that smaller convolution kernels are suitable for extracting detail features and local textures, while larger convolution kernels are more suitable for extracting global image information. Therefore, in the global feature fusion unit, local and global features are combined at the same time to make up for the problem of insufficient image global information extraction capability caused by insufficient receptive field of convolution kernels. The structure of the global feature fusion unit is as follows: Figure 5 shown.

[0042] Taking the second global feature fusion unit as an example, Figure 5 middle, and They are the first concatenated feature and the second concatenated feature after the 5×5 convolution operation of the fourth convolution block, is the fusion sub-feature output by the previous fusion unit, is the intermediate fusion sub-feature, is the output fusion feature. In this unit, the input feature and Use the same structure as the local feature fusion unit to generate intermediate fusion sub-features , and then the fused sub-features of the previous fusion unit are connected and fused, as shown in the above formula (6). Finally, the fused features output by the second global feature fusion unit are input into the decoder to complete image reconstruction.

[0043] In some embodiments of the present invention, see Figure 1, the decoder includes 4 first convolution blocks and 1 fifth convolution block connected in sequence, the first convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fifth convolution block includes a 3×3 convolution layer and a Tanh activation layer connected in sequence.

[0044] in addition, Figure 6 Three examples of obtaining fused images using the image fusion method for AC filter detection according to an embodiment of the present invention are shown. Figure 6 ,Compared with infrared images and visible light images, the fused image can contain more useful ,information, which helps to improve the efficiency of subsequent AC filter ,detection (e.g., fault detection).

[0045] Based on the image fusion method for AC filter detection in the above embodiment, the present invention proposes a computer-readable storage medium.

[0046] In this embodiment, a computer program may be stored on a computer storage medium. When the computer program is executed by a processor, the image fusion method for AC filter detection of the above embodiment is implemented.

[0047] In summary, the image fusion method and storage medium for AC filter detection of the embodiment of the present invention can effectively fuse the infrared image and visible light image of the AC filter by using the fusion model of multi-scale fusion of joint mutual information, wherein the mutual information extraction module can fully learn the complementary characteristics of the thermal distribution state and detail information of the input image, and the multi-scale fusion module can fully capture the local and global information of the image. The specific details are as follows: 1) The infrared image and visible light image of the AC filter are fused using a fusion model of multi-scale fusion of joint mutual information. The fusion model adopts an encoder and decoder structure, in which an independent dual encoder structure is designed for the encoder part to extract the feature information of the two modalities. At the same time, the complementarity between the modalities is enhanced by continuously injecting the mutual information extraction module into the branch network; the features output by the two branch networks are effectively fused through the multi-scale fusion module and input into the decoder to complete the fusion image reconstruction.

[0048] 2) The complementary information of the AC filter infrared image and the visible light image is captured through the mutual information extraction module. Based on the learning of the complementary characteristics of the modalities, this module introduces the channel attention mechanism to accurately screen the important features related to the target task, reducing the interference of irrelevant information on the fusion effect; the extracted complementary information is then re-injected into the features of each modality, enhancing the correlation between the two modalities.

[0049] 3) The multi-scale fusion module is used to capture the local and global feature information of the image through convolution kernels of different sizes. Subsequently, by designing the local and global feature fusion unit, the features of different scales are fused to fully capture the local details and global information and improve the fusion effect.

[0050] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0051] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0052] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0053] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0055] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0057] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. An image fusion method for AC filter detection, characterized in that: include: Acquire infrared and visible light images of the AC filter; Inputting the infrared image and the visible light image into a pre-trained fusion model, and outputting a fusion image for AC filter detection; Among them, the fusion model includes an encoder and a decoder, the encoder includes a first branch network, a second branch network, a mutual information extraction module and a multi-scale fusion module, the first branch network is used to extract the first feature of the infrared image, the second branch network is used to extract the second feature of the visible light image, the mutual information extraction module is used to obtain complementary information according to the first feature and the second feature, and use the channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network respectively, so as to respectively splice with the first feature and the second feature to obtain a first splicing feature and a second splicing feature, the multi-scale fusion module is used to perform multi-scale fusion on the first splicing feature and the second splicing feature to obtain a fused feature, and the decoder is used to decode the fused feature to obtain a fused image.

2. The image fusion method for AC filter detection according to claim 1, characterized in that: The first branch network and the second branch network have the same structure, including four first convolution blocks connected in sequence, and the number of the mutual information extraction modules is three, corresponding one to one with the three first convolution blocks that follow; Among them, the mutual information extraction module is used to obtain complementary information according to the first feature and the second feature output by the corresponding first convolutional blocks in the two branch networks, and use the channel attention mechanism to filter out irrelevant information in the complementary information, and inject the filtered complementary information into the first branch network and the second branch network respectively, so as to perform splicing operations with the first feature and the second feature respectively.

3. The image fusion method for AC filter detection according to claim 2, characterized in that: The mutual information extraction module obtains the features injected into the first branch network through the following formula: in, , represents the features containing shared information, , represents the weight, , Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the first branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

4. The image fusion method for AC filter detection according to claim 2, characterized in that: The mutual information extraction module obtains the features injected into the second branch network through the following formula: in, , represents the features containing shared information, , represents the weight, , Represents the first feature and the second feature of the first convolutional block input in the two-branch network. represents the characteristics injected into the second branch network, and Represent 1×1 and 3×3 convolution operations respectively, Indicates a connection operation. represents the channel attention mechanism, Represents a global average pooling operation.

5. The image fusion method for AC filter detection according to claim 1, characterized in that: The multi-scale fusion module includes a third branch network, a fourth branch network, and a local feature fusion unit, a first global feature fusion unit, and a second global feature fusion unit connected in sequence. The third branch network and the fourth branch network have the same structure, including a second convolution block, a third convolution block, and a fourth convolution block; wherein, The local feature fusion module is used to perform local feature fusion on the first splicing feature and the second splicing feature processed by the second convolution block to obtain a first fused sub-feature; The first global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the third convolution block, and the first fused sub-feature to obtain a second fused sub-feature; The second global feature fusion unit is used to perform global feature fusion on the first splicing feature and the second splicing feature processed by the fourth convolution block, and the second fused sub-feature to obtain the fused feature.

6. The image fusion method for AC filter detection according to claim 5, characterized in that: The local feature fusion unit obtains the first fusion sub-feature by the following formula: in, and represent the first splicing feature and the second splicing feature after being processed by the second convolution block, respectively, represents the first fusion sub-feature, Indicates a connection operation. and Represent 1×1 and 3×3 convolution operations respectively.

7. The image fusion method for AC filter detection according to claim 5, characterized in that: The first global feature fusion unit and the second global feature fusion unit have the same structure. The second global feature fusion unit obtains the second fusion sub-feature by the following formula: in, and Respectively represent the first splicing feature and the second splicing feature after being processed by the third convolution block, represents the second fusion sub-characteristic, , represents the intermediate fusion sub-feature, represents the fusion feature.

8. The image fusion method for AC filter detection according to claim 5, characterized in that: The second convolution block includes a 1×1 convolution layer and a LeakyReLU activation layer connected in sequence, the third convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fourth convolution block includes a 5×5 convolution layer and a LeakyReLU activation layer.

9. The image fusion method for AC filter detection according to claim 1, characterized in that: The decoder includes four first convolution blocks and one fifth convolution block connected in sequence, the first convolution block includes a 3×3 convolution layer and a LeakyReLU activation layer connected in sequence, and the fifth convolution block includes a 3×3 convolution layer and a Tanh activation layer connected in sequence.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image fusion method for AC filter detection according to any one of claims 1 to 9 is implemented.

Images