Infrared and visible image fusion method based on spatial correlation attention
By using a spatial correlation attention-based approach, the problem of fine-grained information loss in the fusion of infrared and visible light images was solved. An encoder and a cross-residual network were used to achieve efficient image feature fusion, thereby improving the fusion effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-06
- Publication Date
- 2026-03-24
AI Technical Summary
Existing methods for fusing infrared and visible light images face challenges in terms of the loss of fine-grained information. Traditional methods lack flexibility, while deep learning-based methods do not perform well in fusing fine-grained information.
A spatial correlation attention-based fusion method is adopted. Multi-scale depth features are extracted by an encoder, and image features are fused using a spatial correlation attention mechanism and a cross residual network. A Unet encoder-decoder network is constructed to reconstruct key information.
It effectively enhances the preservation of fine-grained information during the image fusion process, reduces the risk of information loss, and improves feature utilization and fusion effect.
Smart Images
Figure CN116704274B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a pixel-level fusion method of infrared images and visible light images based on spatial correlation attention, and belongs to the technical field of magnetic image processing. BACKGROUND
[0002] Due to the limitation of imaging mechanism, the information obtained by a single sensor is limited, and the information fusion of multiple imaging sensors is an effective way to obtain good visual expression effect. The infrared image reflects the temperature difference information between scenes, and is less affected by complex environments such as light changes. However, the texture information richness is low. The visible light image reflects the real situation of the environment, and is greatly affected by complex environments such as light changes. However, the detail information is obvious. Therefore, it is of great significance to study the image fusion algorithm of infrared image and visible light.
[0003] In the field of infrared image and visible light image fusion, there are mainly two branches, one is the image fusion based on traditional method, and the other is the image fusion based on deep learning. The fusion strategy of traditional fusion method needs to be designed manually, so its flexibility is poor. The image fusion method based on deep learning has strong representation ability, so it has become the mainstream research direction in the field of image fusion.
[0004] Although the existing image fusion method based on deep learning has obtained valuable fusion results, the loss of fine-grained information in the process of infrared image and visible light image fusion is a challenging task. SUMMARY
[0005] In order to overcome the loss of fine-grained information in the process of existing infrared image and visible light image fusion, the present application provides an infrared and visible light image fusion method based on spatial correlation attention.
[0006] The specific steps of the infrared and visible light image fusion method based on spatial correlation attention are as follows:
[0007] Step one: reading infrared image and visible light image;
[0008] Step two: inputting the infrared image and the visible light image into the encoder to extract the depth features of four scales respectively;
[0009] Step three: inputting the depth features of the infrared image and the depth features of the visible light image into the fusion network;
[0010] Step four: in the fusion network, the corresponding scale of the infrared image depth feature and the visible light image depth feature input into the fusion network are spliced along the channel dimension;
[0011] Step five: in the fusion network, the spatial correlation attention weight of the spliced depth features of the infrared and visible light images is calculated.
[0012] Step six: In the fusion network, multiply the spatial correlation attention weight with the deep feature of the spliced infrared and visible light image, so that the spatial attention acts on the deep feature of the infrared and visible light image;
[0013] Step seven: In the fusion network, the deep feature of the infrared and visible light image after the action of the spatial attention is fused again by using a cross residual network composed of three convolution layers;
[0014] Step eight: input the fused image feature into the decoder to construct the fused image.
[0015] In step two, the encoder is composed of four encoding blocks connected in series, and an image feature is output after each encoding block. Each encoding block is composed of three convolutional and activation layers. The convolution kernel size is 3*3, and the activation function of the activation layer is the Swish activation function.
[0016] In step two, the four layers of single modal image features can be represented as {f1, f2, f3, f4}, where f1 is the deepest feature, f4 is the shallowest feature, the length and width of f4 is 1 / 2 of the input image, the length and width of f3 is 1 / 2 of f4, and so on.
[0017] In step two, the encoder for extracting infrared image features has the same structure and parameters as the encoder for extracting visible light image features.
[0018] In step three, the infrared and visible light image features of different scales output by the encoder are fused respectively, and the fusion network has the same structure.
[0019] In step five, the calculation method of the spatial attention mechanism is as follows:
[0020] For infrared or visible light image, given the intermediate feature map As input (C is the number of feature channels, H is the height of the feature map, and W is the width of the feature map), use θ(·), ψ(·) function to construct strong connections within different scales of local regions and aggregate channel information. θ(·), ψ(·) are convolution functions with convolution kernel size of 1, 3 and 5 respectively, and After dimension transformation of θ(·), ψ(·), the following can be obtained
[0021] Therefore, the correlation function between cross-scale feature elements is defined as:
[0022]
[0023] fi and f j The i and j distribution in represents the position i and j in the feature map f. For formula (1) can be simplified as:
[0024]
[0025] Therefore, the spatial correlation function of the feature in the single modal image can be defined as:
[0026]
[0027] Where C is a normalization factor, and the function g is used to calculate the representation of the input signal at position j.
[0028] The function g is defined as:
[0029]
[0030] After obtaining the spatial correlation of the single modal image feature, the correlation attention mechanism of different modalities is fused to obtain the spatial correlation attention of the mixed modal. Therefore, the definition of the spatial correlation attention is:
[0031] S = δ (Concat{r ir ,r vi ,(g ir (f)+g vi (f))})
[0032] Where δ (·) is a convolution calculation with a convolution kernel size of 1, and Concat represents concatenation along the channel dimension.
[0033] In step eight, the decoder is composed of four decoding blocks, and each decoding block is composed of three convolution activation layers, wherein the convolution kernel size is 3*3.
[0034] Compared with the prior art, the beneficial effects of the present application are:
[0035] The present application constructs a spatial correlation attention mechanism to obtain the correlation between the non-local detail information of the source image, thereby enhancing the fine-grained information and reducing the risk of loss of fine-grained information in the image fusion process.
[0036] The present application constructs a fusion network composed of spatial correlation attention and cross residual structure, wherein the cross residual network constructs the diversity of shallow features and deep features, thereby avoiding the loss of shallow texture information, enriching the transmission mode of information flow and gradient flow, and improving the utilization rate of features.
[0037] The application constructs a Unet-like coding and decoding network, wherein the reconstruction of key information is realized by fusing shallow texture information and deep semantic information in the decoder.
[0038] The application is easy to implement and has good fusion effect. BRIEF DESCRIPTION OF DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0040] Figure 1 The structure diagram of the infrared and visible light image fusion method based on spatial correlation attention of the present application;
[0041] Figure 2 The framework diagram of the spatial correlation attention mechanism;
[0042] Figure 3 The structure diagram of the encoder;
[0043] Figure 4 The spatial distribution diagram of the magnetic anomaly signal;
[0044] Figure 5 The information flow direction diagram of the fused feature and the generated feature;
[0045] Figure 6 The framework diagram of the decoding block. DETAILED DESCRIPTION
[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.
[0047] Referring to Figure 1 The specific steps of the infrared and visible light image fusion method based on spatial correlation attention are as follows:
[0048] Step one: reading the infrared image and the visible light image;
[0049] Step two: input the infrared image and the visible light image into the encoder to extract four scales of deep features; the encoder is composed of four encoding blocks connected in series, and an image feature is output after each encoding block, and each encoding block is composed of three convolutional layers and activation layers. The convolution kernel size is 3*3, and the activation function of the activation layer is the Swish activation function. The four layers of features extracted from a single modal image can be represented as {f1, f2, f3, f4}, where f1 is the deepest layer of features, f4 is the shallowest layer of features, the length and width of f4 are 1 / 2 of the input image, the length and width of f3 are 1 / 2 of f4, and so on. The encoder for extracting infrared image features has the same structure and parameters as the encoder for extracting visible light image features.
[0050] Step three: input the deep features of the infrared image and the deep features of the visible light image into the fusion network; the infrared and visible light image features of different scales output by the encoder are fused respectively, and the fusion network has the same structure.
[0051] Step four: in the fusion network, the infrared image deep features and the visible light image deep features of the corresponding scale input into the fusion network are spliced along the channel dimension.
[0052] Step five: in the fusion network, the spatial correlation attention weight of the spliced deep features of the infrared and visible light images is calculated.
[0053] For an infrared or visible light image, given a middle-level feature map as input (C is the number of feature channels, H is the height of the feature map, and W is the width of the feature map), the θ(·), ψ(·) function is used to construct strong connections within different scales of local regions and aggregate channel information. θ(·), ψ(·) are convolution functions with convolution kernels of 1, 3, and 5, respectively, and After dimension transformation of θ(·), ψ(·), we can get
[0054] Therefore, the correlation function between cross-scale feature elements is defined as:
[0055]
[0056] f i and f j i and j in the distribution represent the position i and j in the feature map f. For formula (1), it can be simplified as:
[0057]
[0058] Therefore, the spatial correlation function of the features in a single modal image can be defined as:
[0059]
[0060] where C is a normalization factor, and the function g is used to calculate the representation of the input signal at position j.
[0061] The function g is defined as:
[0062]
[0063] After obtaining the spatial correlation of single modal image features, the correlation attention mechanisms of different modalities are fused to obtain the spatial correlation attention mechanism of mixed modalities. Therefore, the definition of the spatial correlation attention mechanism is:
[0064] S = δ(Concat{r ir ,r vi ,(g ir (f)+g vi (f))})
[0065] where δ(·) is a convolution calculation with a convolution kernel size of 1, and Concat represents concatenation along the channel dimension.
[0066] Step six: In the fusion network, the spatial correlation attention weight is multiplied by the deep feature after the infrared and visible light image is spliced, so that the spatial attention acts on the deep feature of the infrared and visible light image;
[0067] Step seven: In the fusion network, the deep feature of the infrared and visible light image after the spatial attention is used is fused again by using a cross residual network composed of three convolution layers.
[0068] Step eight: The fused image features are input into the decoder to construct the fused image. The decoder is composed of four decoding blocks, and each decoding block is composed of three convolution activation layers, wherein the convolution kernel size is 3*3.
[0069] In order to better verify the advancement of the method proposed in the present patent, tests were carried out on the MSRS dataset. EN (entropy), MI (mutual information), SF (spatial frequency), AG (average gradient), SD (standard deviation), and VIF (visual information fidelity) were selected as quantitative indicators. The comparison results of the method designed in the present patent and the UMF-CMGR, DIDFuse, RFN-Nest, MEFGAN, and DenseFuse methods are shown in Table 1.
[0070] EN MI SF AG SD VIF UMF-CMGR 5.597 1.92 7.105 2.136 20.754 0.426 DIDFuse 4.207 2.307 9.644 2.013 29.963 0.304 RFN-Nest 6.196 2.448 6.167 2.122 29.088 0.654 MEFGAN 5.138 2.291 7.689 2.21 32.13 0.425 DenseFuse 5.931 2.66 6.021 2.053 23.551 0.692 The method of the invention 6.442 3.157 10.38 3.322 40.677 0.772
[0071] As shown in Table 1, the method designed in the patent application is obviously superior to the other five fusion methods in EN, MI, SF, AG, SD and VIF indexes, indicating that the method designed in the patent application effectively inherits the fine-grained information in the source image and has extremely high popularization value.
[0072] The embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments can be made without departing from the principles and spirits of the present application, and still fall within the protection scope of the present application.
Claims
1. A method for fusing infrared and visible light images based on spatial correlation attention, characterized in that: Includes the following steps: Step 1: Read the infrared and visible light images; Step 2: Input the infrared image and the visible light image into the encoder to extract depth features at four scales; Step 3: Input the depth features of the infrared image and the visible light image into the fusion network; Step 4: In the fusion network, the infrared image depth features and visible light image depth features of the corresponding scale input to the fusion network are stitched together along the channel dimension; Step 5: In the fusion network, spatial correlation attention weights are calculated for the depth features of the stitched infrared and visible light images; Step five specifically includes: The spatial attention mechanism is calculated as follows: For infrared or visible light images, given a mid-level feature map... As input, C is the number of feature channels, H is the height of the feature map, and W is the width of the feature map. , , The function constructs strong connections within local regions at different scales and aggregates channel information. , , These are convolution functions with kernels of 1, 3, and 5, respectively. , , ,right , , After dimensional transformation, we can obtain , , ; The correlation function between feature elements across scales is defined as: ； and The distributions of i and j in the feature map represent the feature map. The positions i and j in the equation can be simplified to: ； The spatial correlation function of features in a single-modal image can be defined as: ； Where C is the normalization factor, and the function g is used to calculate the representation of the input signal at position j. The function g is defined as follows: ； After acquiring the spatial correlation of image features in a single modality, the correlation attention mechanisms of different modalities are fused to obtain a hybrid modality spatial correlation attention mechanism. The definition of the spatial correlation attention mechanism is as follows: ； in For convolution calculation with a kernel size of 1, This represents splicing along the channel dimension; Step 6: In the fusion network, the spatial correlation attention weights are multiplied by the depth features of the stitched infrared and visible light images, so that spatial attention is applied to the depth features of the infrared and visible light images. Step 7: In the fusion network, the depth features of the infrared and visible light images after spatial attention are fused again using a cross residual network consisting of three convolutional layers; in Step 7, the output of the first convolutional layer in the cross residual network is input into the second convolutional layer and added to the output of the second convolutional layer before being input into the third convolutional layer. At the same time, the output of the second convolutional layer is added to the output of the third convolutional layer and used as the final output of the fusion network. Step 8: Input the fused image features into the decoder to construct the fused image.
2. The infrared and visible light image fusion method based on spatial correlation attention according to claim 1, characterized in that: In step two, the encoder consists of four cascaded coding blocks. Each coding block outputs an image feature, and each coding block consists of three convolutional and activation layers.
3. The infrared and visible light image fusion method based on spatial correlation attention according to claim 1 or 2, characterized in that: In step two, the four layers obtained from the single-modality image extraction can be represented as follows: ,in The deepest feature, The shallowest layer features, The length and width are half of the input image. The length and width are 1 / 2, and so on.
4. The infrared and visible light image fusion method based on spatial correlation attention according to claim 1, characterized in that: The encoder used to extract infrared image features in step two has the same structure and parameters as the encoder used to extract visible light image features.
5. The infrared and visible light image fusion method based on spatial correlation attention according to claim 1, characterized in that: In step three, infrared and visible light image features of different scales from the output encoder are fused separately, and the fusion network has the same structure.