SAR and visible light image fusion method based on cross-hybrid attention

Abstract

The present invention discloses a SAR and visible light image fusion method based on cross-hybrid attention. First, the attention mechanism is used as the development point, and then a reasonable attention model is designed to guide the feature recalibration and fusion of deep features of SAR and visible light images in spatial and channel dimensions. The detailed implementation details of the attention model are introduced, and the influence of different pooling operations on the fusion results are analyzed. Finally, experiments are carried out on two groups of real SAR images, and multiple algorithms are used for comparison and quantitative analysis. The experimental results show that the algorithm of the present invention achieves better results in both subjective and objective evaluations. The experimental results verify the effectiveness of the improved network of the present invention in the fusion of SAR and visible light images.

Description

Technical Field

[0001] The present invention relates to the field of image processing methods, and in particular to a SAR and visible light image fusion method based on cross-hybrid attention. Background Art

[0002] Remote sensing image data fusion is the process of processing image data and other information from multiple remote sensors. Synthetic Aperture Radar (SAR) and optical data are the two main types of data from satellite remote sensing platforms. Visible light images can intuitively reflect the true texture, color, and other information of ground objects, but are easily restricted by weather and time, resulting in insufficient data acquisition capabilities. SAR has all-day, all-weather observation capabilities, and microwaves can partially penetrate clouds and sparse vegetation, thereby obtaining rich ground information. Therefore, the fusion of multi-source remote sensing data, such as the fusion between infrared and visible light images, and the fusion of multispectral images and SAR data, has become a research topic that has attracted much attention.

[0003] With the rise of deep learning, numerous deep learning-based fusion algorithms have been proposed. These algorithms extract essential image representations through convolutional neural networks and reconstruct these features to produce fused results. In these deep network-based fusion strategies, only the features in the last layer are used as primary features of the source data. Clearly, during image processing, the rich and effective features in the intermediate layers are lost. Current research indicates that one of the primary goals of multi-source remote sensing image fusion is to preserve spectral information while improving spatial resolution.

[0004] There are many image fusion algorithms, which can generally be divided into the following three categories:

[0005] 1. Image fusion based on traditional methods: Its principle is to use easy-to-implement methods such as transformation and numerical calculation to process each band of the image, such as linear weighting method, high-pass filtering method, IHS transform method, principal component analysis method, etc. 2. Image fusion based on multi-scale: Its basic theory is to parse the image into sub-signals with different frequency characteristics and different resolutions, and then fuse the sub-signals of the corresponding decomposition levels, and reconstruct the fused sub-signals to obtain a fused image, such as the wavelet transform fusion algorithm. 3. Image fusion based on model: The theoretical premise of this fusion method assumes that an image with lower spatial resolution is obtained by downsampling an image with higher resolution or other algorithms that reduce spatial resolution. Based on the above assumptions, by constructing an energy functional representation and a model mapping between high- and low-resolution images, the fused image is finally obtained by solving the optimization model.

[0006] Compared with traditional algorithms, deep learning algorithms are widely used in SAR image fusion. Convolutional Neural Network (CNN) has powerful feature extraction and feature expression capabilities. Therefore, super-resolution image fusion algorithms based on CNN have been proposed one after another. For example, Masi (MASI G, COZZOLINO D, VERDOLIVAL, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594.) first applied super-resolution convolutional neural network (SRCNN) for image pan-sharpening, and the results were significantly better than traditional algorithms; Yang (J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding and J. Paisley, "PanNet: A Deep Network Architecture for Pan-Sharpening," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp.1753-1761, doi:10.1109 / ICCV.2017.193.) et al. proposed the PanNet network, which splits the image fusion task into two parts: spectral preservation and spatial preservation, thereby improving the spectral and spatial quality of the fused image.

[0007] The above fusion algorithm effectively solves the color distortion and blurred spatial details problems found in traditional fusion algorithms. However, it only uses a simple additive fusion strategy for deep features extracted by convolutional neural networks, treating deep features extracted from SAR and visible light images equally. This strategy does not consider the differences in deep features at different spatial locations and in different channels, and therefore cannot fully fuse the deep features of SAR and visible light images. Summary of the Invention

[0008] The present invention provides a SAR and visible light image fusion method based on cross-hybrid attention to solve the problem in the prior art that the convolutional neural network only adopts a simple additive fusion strategy, resulting in the inability to fully fuse the deep features of SAR and visible light images.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] The SAR and visible light image fusion method based on cross-hybrid attention includes the following steps:

[0011] Step 1: Acquire a SAR original image and a visible light original image of the target, preprocess the SAR original image and the visible light original image respectively to obtain a SAR image and a visible light image of the same resolution, and upsample the visible light image to obtain an upsampled visible light image;

[0012] Step 2: extracting multi-scale deep spatial detail features of the SAR image and multi-scale deep spectral preservation features of the visible light image, thereby obtaining a spatial detail feature map of the SAR image and a spectral preservation feature map of the visible light image;

[0013] Step 3: extracting the spatial information weight of the spatial detail feature map of the SAR image obtained in step 2 based on the spatial attention mechanism; then upsampling the visible light image obtained in step 1 to obtain a visible light upsampled image feature map; then, calibrating the visible light upsampled image feature map based on the spatial information weight to obtain a spatially weighted visible light image feature map;

[0014] Extracting a spectral information weight map of the visible light image spectrum preservation feature map obtained in step 2 based on the channel attention mechanism, and calibrating the SAR image spatial detail feature map obtained in step 2 based on the spectral information weight map to obtain a channel-weighted SAR feature map;

[0015] A cascade strategy is used to perform information weighted fusion on the spatially weighted visible light image feature map and the channel-weighted SAR feature map to obtain a fused feature map;

[0016] Step 4: Extract the detailed information of the fused feature map obtained in step 3;

[0017] Step 5: Superimpose the detail information obtained in step 4 on the upsampled image of visible light obtained in step 1 to obtain the final image.

[0018] In the further step 1, the preprocessing of the SAR image includes radiation correction, geometric correction, and speckle filtering in sequence; the preprocessing of the visible light image includes radiation correction, atmospheric correction, resampling, and image registration in sequence.

[0019] In the further step 2, the high-frequency information of the SAR image and the high-frequency information of the visible light image are first extracted respectively; then, the small-scale shallow spatial detail features of the SAR image are extracted based on the high-frequency information of the SAR image, and the large-scale shallow spectral preservation features of the visible light image are extracted based on the high-frequency information of the visible light image; finally, multi-scale feature extraction is performed on the small-scale shallow spatial detail features of the SAR image and the large-scale shallow spectral preservation features of the visible light image, respectively, to obtain the multi-scale deep spatial detail features of the SAR image and the multi-scale deep spectral preservation features of the visible light image, thereby obtaining the spatial detail feature map of the SAR image and the spectral preservation feature map of the visible light image.

[0020] In a further step 2, a high-pass filter is used to obtain high-frequency information of the SAR image and the visible light image, and up-sampling is performed to make the high-frequency information of the SAR image and the visible light image have the same resolution.

[0021] In the further step 2, a separable convolution kernel is used to extract the high-frequency information of the SAR image to obtain the small-scale shallow spatial detail features of the SAR image.

[0022] In the further step 2, a 9×9 convolution kernel is used from the RGB channels to extract the high-frequency information of the visible light image to obtain the large-scale shallow spectrum preservation features of the visible light image.

[0023] Furthermore, in the spatial attention mechanism of step 3, a 1×1 convolution layer is first used to reduce the dimension of the spatial detail feature map of the SAR image along the channel direction; then the softmax function is used to normalize the spatial detail feature map of the SAR image after the dimension reduction to obtain the spatial information weight; then the transposed convolution is used to upsample the visible light image in step 1 to obtain a visible light upsampled image feature map; finally, the spatial information weight is multiplied by the visible light upsampled image feature map to complete the calibration and obtain the spatially weighted visible light image feature map.

[0024] In the further step 3, the visible light image spectral preservation feature map obtained in step 2 is first subjected to global pooling processing and SoftMax function processing based on the channel attention mechanism to generate the spectral information weight map; finally, the spectral information weight map is multiplied by the SAR image spatial detail feature map obtained in step 2 to complete the calibration and obtain the channel-weighted SAR feature map.

[0025] In the further step 4, four convolutional layers are used to extract the detailed information of the fused feature map obtained in step 3, where:

[0026] The first convolution layer is composed of 3×3 convolution kernels, and the first convolution layer generates a 112-channel feature map based on the fused feature map;

[0027] The second convolution layer is composed of 3×3 convolution kernels, which generates a 64-channel feature map based on the output of the first convolution kernel.

[0028] The third convolution layer is composed of 3×3 convolution kernels, which generates a 32-channel feature map based on the output of the second convolution kernel.

[0029] The fourth convolution layer is composed of 3×3 convolution kernels. The fourth convolution kernel reduces the feature channel dimension of the output result of the third convolution kernel to 3 dimensions and outputs the detailed information of the fused feature map.

[0030] Furthermore, in step 5, the detail information obtained in step 4 is superimposed on the upsampled image of visible light obtained in step 1 by using a jumper connection method to obtain a final image.

[0031] This paper proposes a method for extracting deep features from SAR and visible light images. By combining spatial and channel attention mechanisms in a convolutional neural network, different weights are assigned to points at different spatial locations and feature maps of different channels in the deep features to achieve adaptive recalibration of the deep features. This allows for better fusion of the deep features, further enhancing the spatial detail and spectral information of the fused image, and improving the fusion quality. Compared with the prior art, the present invention has the following advantages:

[0032] (1) In the proposed fusion framework, the actual spectral information of the visible light image is used to correct the feature map of the SAR image, and the high-precision spatial detail information of the SAR image is used to enhance the spatial information of the visible light image.

[0033] (2) Compared with the existing technology, this convolutional neural network combined with the attention mechanism can autonomously focus on information that is useful for the result and ignore useless information, greatly enhancing the network's ability to extract features; the attention mechanism can effectively recalibrate the deep features of the image in the spatial dimension, channel dimension, and gradient domain dimension, emphasizing features that are useful for the task and suppressing features that are useless for the task, thereby improving the accuracy of the final result. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the SAR data preprocessing step in the first embodiment of the present invention.

[0035] Figure 2 This is the visible light data preprocessing step in the first embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the feature fusion solution (CroAM) based on the hybrid attention model in Example 1 of the present invention.

[0037] Figure 4This is a schematic diagram of a method according to a first embodiment of the present invention.

[0038] Figure 5 This is the quantitative result of the fusion of the dataset near Tianjin Lele Island in Example 2 of the present invention.

[0039] (a) is a visible light image near Tianjin Lele Island; (b) is a SAR image; (c) is an IHS fusion image; (d) is the RSIFNN fusion image; (e) is a Wavelet image; (f) is an NSCT-AVG image; (g) is an IHS-NSST-SR image; (h) is a dual-branch method image; (i) is a result image of the method of the present invention; (j) is a visible light local image; (k) is a dual-branch method local image; (l) is a local image of the method of the present invention.

[0040] Figure 6 This is the quantitative result of the fusion of the Bohai Port dataset in Example 2 of the present invention. Among them: (a) is the visible light image of Bohai Port; (b) is the SAR image; (c) is the IHS fusion image; (d) is the RSIFNN fusion image; (e) is the Wavelet image; (f) is the NSCT-AVG image; (g) is the IHS-NSST-SR image; (h) is the dual-branch method image; (i) is the result of the method of the present invention; Figure 6 j is a local diagram of visible light; (k) is a local diagram of the dual-branch method; (l) is a local diagram of the method of the present invention. DETAILED DESCRIPTION

[0041] To help those skilled in the art better understand the present invention, the following detailed description of the embodiments of the present invention is provided in conjunction with the accompanying drawings and examples. This will help those skilled in the art to fully understand and implement the present invention by applying technical means to solve technical problems and achieve corresponding technical effects. The embodiments of the present invention and the various features therein may be combined with each other as long as they do not conflict with each other, and the resulting technical solutions are all within the scope of protection of the present invention.

[0042] Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0044] Example 1

[0045] This embodiment discloses a SAR and visible light image fusion method based on cross-hybrid attention, which is implemented based on an improved convolutional neural network. In this embodiment, a high-pass filter, an encoder layer, a CroAM feature fusion layer, and a decoder layer are added to the existing convolutional neural network CNN architecture to obtain an improved convolutional neural network. Its architecture is as follows: Figure 4 As shown. Among them, the high-pass filter is used to extract high-frequency information in step 2 of the embodiment method. The encoder layer includes a separable convolution kernel, a 9×9 convolution kernel, and a multi-scale feature extraction module, which is used as a feature extraction module to perform feature extraction in step 2 of the method of this embodiment. The CroAM feature fusion layer includes a spatial attention mechanism module and a channel attention mechanism module, which is used to implement feature fusion based on cross-mixed attention in step 3 of the method of this embodiment. The decoder layer includes four convolution layers, each of which is composed of a 3×3 convolution kernel, which is used to extract detail information in step 4 of the method of this embodiment.

[0046] In this embodiment, the method of this embodiment is described in conjunction with the improved convolutional neural network used. The method of this embodiment includes the following steps:

[0047] Step 1: Obtain a SAR raw image and a visible light raw image of the target. Preprocess the SAR raw image and the visible light raw image to obtain SAR images and visible light images of equal resolution. Then, upsample the visible light image by a factor of 3 using bicubic interpolation to obtain an upsampled visible light image.

[0048] Because SAR and visible light images have different imaging mechanisms, this embodiment requires preprocessing the original SAR and visible light images separately before data fusion. Step 1 of this embodiment is implemented based on the preprocessing module of the input layer of the existing convolutional neural network (CNN) and the upsampling module of the existing convolutional neural network (CNN).

[0049] like Figure 1 、 Figure 2 As shown in the figure, the preprocessing of SAR images includes radiometric correction, geometric correction, and speckle filtering, while the preprocessing of visible light images includes radiometric correction, atmospheric correction, resampling, and image registration. After preprocessing, the resulting SAR and visible light images have the same resolution.

[0050] Step 2: In this embodiment, a high-pass filter is used to extract the high-frequency information of the SAR image and the high-frequency information of the visible light image obtained in step 1, and upsampling is performed to make the high-frequency information of the SAR image and the visible light image have the same resolution.

[0051] Considering the feature differences between SAR and visible light satellite sensor images, this embodiment uses two different feature extraction modules at the encoder layer to extract spatial detail features from the high-frequency information of SAR images and spectral preservation features from the high-frequency information of visible light images.

[0052] The separable convolution kernel in the encoder layer is used to extract the small-scale shallow spatial detail features of the SAR image based on the high-frequency information of the SAR image. It has the advantage of better response to smaller-scale features and can better extract the feature information of high-resolution SAR images.

[0053] The 9×9 convolution kernel in the encoder layer is used to extract the high-frequency information of the visible light image to obtain the large-scale shallow spectral preservation features of the visible light image, which has the advantages of better responding to coarser features and better extracting the feature information of low-resolution visible light images.

[0054] This embodiment then uses a multiscale feature extraction module to extract small-scale shallow spatial detail features from SAR images and large-scale shallow spectral preservation features from visible light images. This results in multiscale deep spatial detail features from SAR images and multiscale deep spectral preservation features from visible light images, thereby obtaining spatial detail feature maps from SAR images and spectral preservation feature maps from visible light images. The multiscale feature extraction module flexibly and effectively integrates the extracted features explicitly into the network.

[0055] Step 3. This embodiment uses the CroAM feature fusion layer to perform cross-hybrid attention-based feature fusion on the spatial detail feature map of the SAR image and the spectral preservation feature map of the visible light image obtained in Step 2. By cross-hybridizing the spatial attention mechanism and the channel attention mechanism in the CroAM feature fusion layer, different weights are assigned to points at different spatial locations in the deep features and feature maps of different channels to achieve adaptive recalibration of the deep features, thereby better fusing the deep features, further enhancing the spatial detail information and spectral information of the fused image, and improving the fusion quality.

[0056] In computer vision, the core idea of ​​the attention mechanism is to enable the network to autonomously focus on important information and ignore irrelevant information. In recent years, the attention mechanism has been introduced to deep learning-based applications such as natural language processing, image super-resolution, image classification, and image fusion, achieving tremendous success. Neural networks incorporating this attention mechanism can autonomously focus on information that contributes to the result while ignoring irrelevant information, significantly enhancing the network's ability to extract features.

[0057] Attention mechanisms are divided into soft attention and hard attention according to the differentiability of attention. Soft attention focuses more on spatial or channel dimension information, and autonomously learns the attention weight map through the network's forward propagation and backward feedback. Hard attention, on the other hand, is a 0 / 1 problem, determining which areas are attended to and which are not. The most notable feature of hard attention is that it is not differentiable, so the weight map cannot be autonomously learned through the network's backward propagation algorithm. The training process is often completed through reinforcement learning. Soft attention mechanisms are often used in the field of image processing. Typical networks include: spatial transformer networks, SENet, CBAM, and DECA. A new attention mechanism network, SENet, is proposed in the literature (Hu J, Shen L, Albanie S, et al. Squeeze-and-Excitation Networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 42 (8): 2011-2023.). In this paper, the network automatically obtains the weight map of the importance of each feature channel through learning, and then selectively enhances useful channel features and suppresses useless channel features according to this weight map, thereby achieving adaptive correction of feature channels.

[0058] The core steps of SENet include three parts: squeeze, excitation, and reweighting (scale attention). The squeeze operation uses global average pooling to transform the H×W×C feature map into a 1×1×C real number sequence, where H represents the width of the image, W represents the height of the image, and C represents the number of channels. The excitation operation is implemented through a gating mechanism and a softmax activation function to fully capture the dependencies between channels. Reweighting multiplies the excitation channel weights by the corresponding feature channels to achieve adaptive adjustment of the original features in the channel dimension.

[0059] In the paper (Woo S, Park J, Lee JY, et al. CBAM: Convolutional Block Attention Module [C] / / 15th European Conference on Computer Vision. Munich: Springer-Verlag, 2018: 3-19), a simple and effective attention module, the Convolutional Block Attention Module (CBAM), is proposed. CBAM employs global max pooling and global average pooling in the channel dimension, resulting in finer-grained attention than SENet. Furthermore, CBAM further improves network performance by adding spatial attention. Ultimately, the CBAM module learns which channels and locations to effectively emphasize or suppress intermediate features. To validate its effectiveness, the paper also conducts experiments on the ImageNet-1K, MSCOCO, and VOC2007 datasets, demonstrating consistent improvements in classification and object detection.

[0060] In the paper (Cui Y, Du H, Mei W. Infrared and Visible Image Fusion Using Detail Enhanced Channel Attention Network [J]. IEEE Access, 2019, 7:182185-182197.), a detail-enhanced channel attention module (DECA) is proposed and applied to the fusion of infrared and visible light images. The DECA module adaptively recalibrates the feature maps using the average value of each channel in the feature map in the image brightness and gradient domains, thereby emphasizing useful features through learning. Experimental results show that the DECA module significantly increases the detail information of the infrared and visible light fusion image.

[0061] Based on the above research, it can be seen that the attention mechanism can effectively recalibrate the deep features of the image in the spatial dimension, channel dimension and gradient domain dimension, emphasize the features that are useful for the task, and suppress the features that are useless for the task, thereby improving the accuracy of the final result.

[0062] Therefore, this embodiment adopts a cross-hybrid attention mechanism to utilize the complementary information of SAR image data and visible light image data, and designs a CroAM feature fusion layer based on the cross-hybrid attention model. It utilizes the respective advantages of the deep features of SAR image data and visible light image data to enhance the spectral information and spatial detail texture information of the feature map generated by the feature extraction module.

[0063] In the CroAM feature fusion layer of this embodiment, the spatial information weights of the SAR image spatial detail feature map obtained in step 2 are extracted through the spatial attention mechanism module based on the spatial attention mechanism, and the visible light image in step 1 is upsampled to obtain a visible light upsampled image feature map. The spectral information weights of the visible light image spectral preservation feature map obtained in step 2 are extracted through the channel attention module based on the channel attention mechanism. Then, based on the spatial information weights, the visible light upsampled image feature map is calibrated to obtain a spatially weighted visible light image feature map; and based on the spectral information weights, the SAR image spatial detail feature map obtained in step 2 is calibrated to obtain a channel-weighted SAR feature map. Finally, a cascade strategy is used to fuse the spatially weighted feature map and the channel-weighted feature map to obtain a fused feature map.

[0064] The specific process of feature description performed by the CroAM feature fusion layer in this embodiment is as follows:

[0065] 1. In the spatial attention mechanism module of this embodiment, attention is paid to the important points of deep features in the spatial dimension and fusion is performed. In the spatial attention mechanism module of this embodiment, a 1×1 convolution is first used to calculate the value of the H×W×N (H represents the width of the image, W represents the height of the image, and C represents the number of channels of the image) SAR image spatial detail feature map at each point in space by performing L1 norm, that is, the channel dimension summation form is compressed into one dimension to obtain the H×W×1 matrix value. Then the matrix value is operated by the SoftMax function to extract the spatial information weight map. The visible light image of step 1 is upsampled by transposed convolution to obtain the visible light upsampled image feature map, the SAR spatial information weight map is multiplied with the visible light upsampled image feature map, the spatial dimension recalibration of the visible light upsampled image feature map is completed, and finally the re-adjusted spatial weighted visible light image feature map is added to achieve feature fusion. The specific process can be shown by formulas (1), (2), and (3):

[0066]

[0067] Φ sk (x,y)=α k (x,y)×Φ k (x,y) (2),

[0068]

[0069] In formula (1)-formula (3), Φ k (x, y) represents the value of the deep feature map of the k-th input image at the spatial position (x, y), which is a vector of N-dimensional channels, where k∈1,...K and K=2. Specifically, in this embodiment, k takes the values ​​1 and 2, representing the input SAR and visible light images, respectively.

[0070] x and y are the coordinates of the SAR image spatial detail feature map, and the values ​​of x and y are within the range of the length and width of the SAR image spatial detail feature map.

[0071] ||·||1 indicates that the spatial detail feature map of the multi-channel SAR image is summed according to the channel dimension at a certain point, that is, the N-dimensional vector is summed.

[0072] α k (x,y) represents the spatial information weight of the spatial detail feature map of the SAR image obtained by the SoftMax method. The specific SoftMax operation is: input multiple values ​​and calculate the ratio of a certain value to the sum of all values.

[0073] Φ sk (x, y) represents the recalibrated spatially weighted visible light image feature map obtained by multiplying the spatial information weight of the SAR image spatial detail feature map by the second visible light upsampled image feature map.

[0074] Φ sf (x,y) represents the fusion result of the spatially weighted visible light image feature map after superposition and recalibration.

[0075] The s in the subscripts sk and sf represents the spatial dimension of the feature.

[0076] Compared to visible light image data, SAR image data has higher accuracy in spatial information. Leveraging its high-precision spatial information to guide and weight feature maps extracted from visible light images can make the spatial features extracted from visible light image data more accurate and reliable. Therefore, this embodiment uses a spatial attention mechanism to extract spatial information weights from SAR image spatial detail feature maps and uses these spatial information weights to recalibrate the visible light image spectrum-preserving feature maps. This introduces spatial information from SAR image data to enhance visible light image data.

[0077] 2. First, based on the channel attention mechanism, the visible light image spectrum preservation feature map obtained in step 2 is used to generate the spectral information weight; finally, the spectral information weight is multiplied by the spatial detail feature map of the SAR image extracted in step 2 to obtain the channel-weighted SAR feature map.

[0078] In the existing deep learning-based fusion methods, most fusion strategies only calculate the spatial dimension information of the features, but the deep features are three-dimensional tensors. Therefore, in the fusion strategy, this embodiment also needs to consider the differences in the feature channel dimension information. Combined with the channel attention mechanism. The channel attention mechanism of this embodiment focuses on the saliency feature map on the channel dimension in the deep features and performs fusion. This embodiment first compresses the feature space dimension of the visible light image spectrum preservation feature map obtained in step 2 by a global pooling operation to obtain a 1×1×N series value, and then calculates the spectral information weight map through the SoftMax operation, and multiplies the spectral information weight map with the spatial detail feature map of the SAR image to obtain a channel-weighted SAR feature map with the channel dimension readjusted. Finally, the adjusted channel-weighted SAR feature map and the spatially weighted visible light image feature map are added together for cascade fusion. The specific process can be expressed by formulas (4), (5), and (6):

[0079] β k (m)=P(Φ k (m)) (4),

[0080]

[0081]

[0082] In formula (4)-formula (6), Φ k (m) represents the mth channel feature map in the deep features of the kth input image. In this embodiment, k takes the values ​​1 and 2, representing the input SAR and visible light images, respectively.

[0083] P(·) represents the global pooling operation on the spectrum-preserving feature map of the visible light image.

[0084] β k (m) represents the 1×1×N sequence value after global pooling. In the channel attention mechanism of this embodiment, two global pooling operators are selected: (a) average operator: calculates the average value of each channel; (b) maximum operator: calculates the maximum value of each channel.

[0085] β ck (m) represents the spectral information weight map obtained after the SoftMax operation on the sequence value;

[0086] β cf (m) represents the result of multiplying and adding k deep features with the spectral information weight map, that is, the fused channel-weighted SAR feature map, where the subscript c represents the channel dimension of the feature.

[0087] Compared to SAR image data, visible light image data contains richer contour and spectral information about ground objects. If visible light image data is directly fused with SAR image data, the resulting visible light image will clearly have lower reliability in terms of spectral information. Therefore, this embodiment utilizes the actual spectral information of the visible light image data to guide the generation of SAR image data feature maps, ensuring that the feature maps extracted from the SAR image data simultaneously possess high-precision contour, spectral, and spatial texture information. Therefore, in this embodiment, the spectral information weights of the visible light image spectral-preserving feature maps are extracted based on a channel attention mechanism, and the spatial detail feature maps of the SAR image are recalibrated. This introduces the spectral information of the visible light image data to enhance the SAR image data.

[0088] It should be noted that, unlike the existing channel attention mechanism, the channel attention mechanism of this embodiment takes into account the complex data distribution of SAR data and visible light data. Therefore, this embodiment deletes the pooling operation widely used in the existing channel attention mechanism to avoid numerical anomalies caused by mean pooling or maximum pooling. In the channel attention mechanism of this embodiment, the visible light image spectral preservation feature map obtained in step 2 is first compressed by a global pooling operation to obtain a 1×1×N series value, and then normalized directly using the softmax function to obtain the spectral information weight; Subsequently, this embodiment multiplies the spectral information weight with the SAR image spatial detail feature map extracted in step 2 to obtain a feature map of the channel-weighted SAR image.

[0089] Furthermore, unlike the global channel attention weights used in existing channel attention mechanisms, this embodiment uses local weights, which can better adapt to the large dynamic range of SAR image data. A cascade strategy is used to perform information-weighted fusion of the spatially weighted visible light image feature map and the channel-weighted SAR feature map to obtain a fused feature map.

[0090] 3. In order to treat the spatially weighted visible light image feature maps and channel-weighted SAR feature maps extracted by the two attention mechanisms equally, this embodiment uses a cascade strategy to fuse the spatially weighted visible light image feature maps and channel-weighted SAR feature maps obtained by the two modules. Combining the spatial and channel attention mechanisms, the deep features are first recalibrated through the two attention mechanisms, and then the output of the weighted fusion attention model is used to obtain the final fusion feature. The fusion process based on the attention mechanism is as follows: Figure 3 shown. Figure 3 In the equation, Φ1 and Φ2 represent the deep feature extraction blocks of SAR and visible light images respectively (i.e., the spatial detail feature map of SAR image and the spectrum preservation feature map of visible light image). s and Φ vRespectively represent the results of recalibrating and superimposing the features in the spatial and channel dimensions through the spatial attention and channel attention mechanisms, and finally the Φ output by the attention mechanism is s and Φ v The weighted fusion process is used to obtain the fusion feature map. The process can be expressed by formula (7):

[0091] Φ f =(Φ s+ Φ v )*0.5 (7).

[0092] Step 4: At the decoder layer, a four-layer convolutional layer is used to extract the detailed information of the fused feature map obtained in step 3, where:

[0093] The first convolution layer is composed of 3×3 convolution kernels, and the first convolution layer generates a 112-channel feature map based on the fusion feature map;

[0094] The second convolution layer is composed of 3×3 convolution kernels, which generates a 64-channel feature map based on the output of the first convolution kernel.

[0095] The third convolution layer is composed of 3×3 convolution kernels, which generates a 32-channel feature map based on the output of the second convolution kernel.

[0096] The fourth convolution layer is composed of 3×3 convolution kernels. The fourth convolution kernel reduces the feature channel dimension of the output result of the third convolution kernel to 3 dimensions and outputs clear detailed information of the fusion feature map.

[0097] Step 5: Superimpose the detail information obtained in step 4 onto the upsampled image of visible light obtained in step 1 to obtain the final image.

[0098] Specifically, a jumper connection method is used to superimpose the detailed information of the fused feature map output by the CroAM feature fusion layer branch onto the upsampled image of visible light obtained in step 1 to obtain the final image. That is, the spectral information of visible light is transferred to the fused image. The implementation process is shown in formula (8):

[0099] F=F Cro ⊕↑F vis (8),

[0100] In formula (9): F Cro Indicates the detailed information of the fused feature map obtained in step 4, ↑F vis Represents the upsampled image of visible light obtained by 3x upsampling using bicubic interpolation in step 1.

[0101] Example 2

[0102] This example discloses the experimental results and analysis of the fusion method described in Example 1, which are specifically described as follows:

[0103] 1. Experimental data and parameter settings

[0104] The experiment used two sets of real SAR data. The training data consisted of Sentinel-1 sub-pixel SAR images and Landsat-8 multispectral images. After image preprocessing, 9,600 SAR and visible light image pairs were cropped, 90% of which were used for training and 10% for validation. The test data consisted of two sets. The first set consisted of Sentinel-1 SAR images and Landsat-8 visible light images of the area near Lele Island, Tianjin, containing a rich variety of features such as buildings, water bodies, and farmland. The second set covered the Bohai Sea port area, which contains numerous features such as crops and bridges. The SAR images also contained target information that was not detectable in visible light images. Detailed information about the SAR and visible light datasets is shown in Table 1.

[0105] Table 1 Prior information of SAR and visible light image information

[0106]

[0107] The parameters of the algorithm proposed in this embodiment are set as follows:

[0108] The experimental environment for this example is Windows 10, 64-bit operating system, NVIDIA Quadro M2000-4G graphics card, and a two-branch convolutional neural network built under the TensorFlow framework. The training batch size is 100, the number of iterations is 30,000, and the optimizer is Adam, with a learning rate of 0.001 and a momentum decay coefficient of 0.9.

[0109] In order to verify the effectiveness of the fusion method described in Example 1 in terms of spectrum preservation and detail enhancement, this example selects the traditional methods IHS, Wavelet, IHS_NSST_SR, NSCT_AVG and the deep learning method RSIFNN as the comparative experimental group.

[0110] 2. Experimental comparative analysis

[0111] (a) Pooling operation experiment

[0112] In the above channel attention model, two different pooling operators are used to perform global pooling operations on the channel dimension of the image depth feature. The experimental results are shown in Table 2 below.

[0113] Table 2. Evaluation results of fused image quality corresponding to different pooling operations

[0114]

[0115] Note: "Average operator" and "Maximum operator" represent different global pooling operators in the channel attention model fusion strategy.

[0116] As shown in Table 2, compared to the average operator in the attention-based fusion strategy, the maximum operator achieves nearly optimal values ​​across all metrics. This demonstrates that the maximum operator is effective in the channel-attention model, capturing more salient features from deep features for image fusion. Therefore, all subsequent experiments in this chapter use the maximum operator for global pooling in the channel dimension.

[0117] (b) Analysis of change detection results of the dataset near Tianjin Lele Island

[0118] The first set of experimental images contains a large number of buildings, farmlands, water bodies and other objects, which are rich in detail information and spectral information. Figure 5 As can be seen from the red box areas in (a) and (b), the islands in the water in the visible light image have low visibility and the targets are not obvious, while the islands in the SAR image have clear outlines and the targets are prominent. Figure 5 (c) and Figure 5 (d) As can be seen, the IHS algorithm and RSIFNN algorithm significantly enhance the spatial information of the visible light image, but there is also obvious color distortion, such as the island below the fused image and the farmland crops in the image. Figure 5 The Wavelet transform algorithm in (e) is better than the IHS algorithm in preserving spectral information, and the spatial detail information is improved to a certain extent, but the aliasing phenomenon is obvious in the fusion result. Figure 5 (f) NSCT_AVG method, there are artifacts at the boundaries of cultivated fields, the details of building areas are blurred, and the fusion is unnatural. Figure 5 (g) The IHS_NSST_SR method shows obvious blockiness in the texture of the building area, and the boundary of the cultivated field is relatively blurred. Figure 5 The color of the fused image by the dual-branch method (h) is closer to the original multispectral image, and the visual effect is better, but the spatial details are still not clear enough.

[0119] Example 1 Fusion Method Figure 5 (i) Compared with other algorithms, it achieves the best fusion effect.

[0120] First, the spectral information of the fused image of the embodiment 1 fusion method is well maintained. Second, the edge contours of islands and buildings in the fused image are enhanced. Compared with the dual-branch method in image 5 (h), the fusion method of embodiment 1 recalibrates the deep features of SAR and visible light images based on the attention model. The network can adaptively learn features that are more meaningful to the fusion task, inject significant spatial detail information in the SAR image into the fused image, and magnify the local details. Figure 5 (j) Figure 5 (k) and Figure 5 It can be seen from the roof in (l) that the clarity gradually increases, indicating that the fusion method in Example 1 further enhances the spatial detail information of the fused image.

[0121] (c) Analysis of change detection results of Bohai Port dataset

[0122] The second set of experimental images contains farmland, houses, bridges and other features. Figure 6 As can be seen from the boxed areas in (a) and (b), there are obvious target information in the SAR image that does not exist in the visible light image, such as bridges and islands. The analysis of the fusion results of each algorithm is basically the same as the above analysis. Figure 6 (j) to Figure 6 (l) The building area in the local image also shows that the fusion method of Example 1 makes the spatial details of the fused image clearer and the image quality is further enhanced. The subjective evaluation shows that the algorithm proposed in this chapter is ideal.

[0123] (d) Objective analysis

[0124] To better evaluate the pros and cons of each fusion algorithm, some evaluation metrics are needed to objectively assess the quality of the fused image. Tables 3 and 4 show the evaluation results of the fusion methods of IHS, RSIFNN, Wavelet, NSCT_AVG, IHS_NSST_SR, dual-branch, and Example 1 on two sets of data.

[0125] Table 3 Evaluation results of the first set of experimental data

[0126]

[0127]

[0128] In Table 3, based on the spectral evaluation indicators CC and RMSE, it can be seen that the IHS method performs the worst, and the principal component replacement method has obvious color distortion; the RSIFNN method extracts too much SAR intensity information, which also leads to color distortion and poor RMSE indicators; the spectral quality of the Wavelet and NSCT_AVG methods has been greatly improved, but there is still room for improvement; the IHS_NSST_SR method performs well in spectral quality, while the dual-branch method and the method of the present invention further improve spectral quality. From the spatial quality evaluation indicators PSNR and SSIM, it can be seen that the PSNR of the IHS and RSIFNN methods is poor, indicating that there is obvious image distortion in the fusion results of these two methods; due to the constraint of the wavelet transform direction, the fusion result of the Wavelet fusion method is jagged and performs poorly in terms of the SSIM indicator; the NSCT_AVG method and the IHS_NSST_SR method have made significant improvements in spatial information preservation, while the PSNR and SSIM indicators of the dual-branch method and the fusion method of Example 1 are further improved.

[0129] The overall evaluation index of spatial and spectral image quality, ERGAS, shows that the poor evaluation indexes of the IHS and RSIFNN methods are due to the presence of color distortion and image distortion in both methods, which is consistent with subjective evaluation. The Wavelet and NSCT_AVG methods have greatly improved. The IHS_NSST_SR method fully combines the advantages of multi-resolution analysis and component replacement methods to improve the overall evaluation index. The comprehensive index of the fusion method in Example 1 is further improved.

[0130] Table 4 Evaluation results of the second set of experimental data

[0131]

[0132]

[0133] In Table 4, the trends in the evaluation indicators for the fusion results of each algorithm are similar to those in Table 3. In summary, the fusion method of Example 1 effectively enhances the spatial detail information of the multispectral image while maintaining spectral information. Furthermore, the fusion method of Example 1 introduces an attention model based on the dual-branch method, further enhancing the spatial detail information and spectral information of the multispectral image. All evaluation indicators of the image are closest to the ideal value.

[0134] In summary, in order to overcome the shortcomings of the deep features extracted by convolutional neural networks, which only adopt a simple additive fusion strategy, do not consider the differences of deep features in different spatial positions and different channels, and cannot fully fuse the deep features of SAR and multispectral images, this paper proposes a SAR and visible light image fusion method based on a hybrid attention convolutional neural network. The characteristics of this method are:

[0135] (1) The method of the present invention introduces a spatial and channel attention model based on the dual-branch convolutional neural network method to further enhance the spatial detail information and spectral information of the visible light image, and various evaluation indicators of the image are closest to the ideal value.

[0136] (2) The method of the present invention has effectively solved the problems of color distortion and blurred spatial details existing in traditional fusion algorithms. For the deep features extracted by the CNN network, only a simple additive fusion strategy is adopted. The method of the present invention has effectively solved this problem, treating the deep features extracted from SAR and visible light images equally, considering the differences of deep features in different spatial positions and different channels, fully fusing the deep features of SAR and visible light images, and improving the fusion quality.

[0137] The preferred embodiments of the present invention are described in detail above with reference to the accompanying drawings. The embodiments described in the present invention are merely descriptions of the preferred embodiments of the present invention and do not limit the concept and scope of the present invention. The various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. Such combinations should also be regarded as the contents disclosed in this disclosure as long as they do not violate the concept of the present invention. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.

[0138] The present invention is not limited to the specific details of the above-mentioned embodiments. Within the scope of the technical concept of the present invention and without departing from the design concept of the present invention, various modifications and improvements made to the technical solution of the present invention by those skilled in the art should fall within the scope of protection of the present invention. The technical contents for which protection is sought in the present invention have been fully recorded in the claims.

Claims

1. A SAR and visible light image fusion method based on cross-hybrid attention, characterized by: The following steps are involved: Step 1: Acquire a SAR original image and a visible light original image of the target, preprocess the SAR original image and the visible light original image respectively to obtain a SAR image and a visible light image of the same resolution, and upsample the visible light image to obtain an upsampled visible light image; Step 2: extracting multi-scale deep spatial detail features of the SAR image and multi-scale deep spectral preservation features of the visible light image, thereby obtaining a spatial detail feature map of the SAR image and a spectral preservation feature map of the visible light image; Step 3: extracting the spatial information weight of the spatial detail feature map of the SAR image obtained in step 2 based on the spatial attention mechanism; then upsampling the visible light image obtained in step 1 to obtain a visible light upsampled image feature map; then, calibrating the visible light upsampled image feature map based on the spatial information weight to obtain a spatially weighted visible light image feature map; Extracting a spectral information weight map of the visible light image spectrum preservation feature map obtained in step 2 based on the channel attention mechanism, and calibrating the SAR image spatial detail feature map obtained in step 2 based on the spectral information weight map to obtain a channel-weighted SAR feature map; A cascade strategy is used to perform information weighted fusion on the spatially weighted visible light image feature map and the channel-weighted SAR feature map to obtain a fused feature map; Step 4: Extract the detailed information of the fused feature map obtained in step 3; Step 5: Superimpose the detail information obtained in step 4 on the upsampled image of visible light obtained in step 1 to obtain the final image.

2. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1 is characterized in that: In step 1, the preprocessing of SAR images includes radiation correction, geometric correction, and speckle filtering in sequence; the preprocessing of visible light images includes radiation correction, atmospheric correction, resampling, and image registration in sequence.

3. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1 is characterized in that: In step 2, the high-frequency information of the SAR image and the high-frequency information of the visible light image are first extracted respectively; then, the small-scale shallow spatial detail features of the SAR image are extracted based on the high-frequency information of the SAR image, and the large-scale shallow spectral preservation features of the visible light image are extracted based on the high-frequency information of the visible light image; finally, multi-scale feature extraction is performed on the small-scale shallow spatial detail features of the SAR image and the large-scale shallow spectral preservation features of the visible light image, respectively, to obtain the multi-scale deep spatial detail features of the SAR image and the multi-scale deep spectral preservation features of the visible light image, thereby obtaining the spatial detail feature map of the SAR image and the spectral preservation feature map of the visible light image.

4. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 3 is characterized in that: In step 2, a high-pass filter is used to obtain the high-frequency information of the SAR image and the visible light image, and upsampling is performed to make the high-frequency information of the SAR image and the visible light image have the same resolution.

5. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 3 is characterized in that: In step 2, a separable convolution kernel is used to extract the high-frequency information of the SAR image to obtain the small-scale shallow spatial detail features of the SAR image.

6. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 3 is characterized in that: In step 2, a 9×9 convolution kernel is used from the RGB channel to extract the high-frequency information of the visible light image to obtain the large-scale shallow spectrum preservation features of the visible light image.

7. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1, characterized in that: In the spatial attention mechanism of step 3, a 1×1 convolution layer is first used to reduce the dimension of the spatial detail feature map of the SAR image along the channel direction; then the softmax function is used to normalize the spatial detail feature map of the SAR image after dimensionality reduction to obtain the spatial information weight; then the visible light image in step 1 is upsampled using transposed convolution to obtain the visible light upsampled image feature map; finally, the spatial information weight is multiplied by the visible light upsampled image feature map to complete the calibration and obtain the spatially weighted visible light image feature map.

8. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1 is characterized in that: In step 3, the visible light image spectral preservation feature map obtained in step 2 is first globally pooled and processed with the SoftMax function based on the channel attention mechanism to generate the spectral information weight map; finally, the spectral information weight map is multiplied with the SAR image spatial detail feature map obtained in step 2 to complete the calibration and obtain the channel-weighted SAR feature map.

9. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1, characterized in that: In step 4, a four-layer convolutional layer is used to extract the detailed information of the fused feature map obtained in step 3, where: The first convolution layer is composed of 3×3 convolution kernels, and the first convolution layer generates a 112-channel feature map based on the fusion feature map; The second convolution layer is composed of 3×3 convolution kernels, which generates a 64-channel feature map based on the output of the first convolution kernel. The third convolution layer is composed of 3×3 convolution kernels, which generates a 32-channel feature map based on the output of the second convolution kernel. The fourth convolution layer is composed of 3×3 convolution kernels. The fourth convolution kernel reduces the feature channel dimension of the output result of the third convolution kernel to 3 dimensions and outputs the detailed information of the fused feature map.

10. The SAR and visible light image fusion method based on cross-hybrid attention according to claim 1, characterized in that: In step 5, the detail information obtained in step 4 is superimposed on the upsampled image of visible light obtained in step 1 by using a jumper connection method to obtain the final image.

Images