A method for fusing visible light images and SAR images

Abstract

The application discloses a visible light image and SAR image fusion method, comprising the following steps: S1: constructing a fusion network based on a Transformer network introducing an LT model and a Dropkey mechanism and a CNN network introducing a CBAM module; S2: inputting visible light images and SAR images of a training set into an encoder to extract global features and local features; S3: fusing and splicing the global features of the two images, inputting the spliced features into a decoder together with the local features of each image, and reconstructing original images; S4: inputting the visible light images and the SAR images of the training set into the encoder trained in step S3 to extract global features and local features; S5: fusing and splicing the global features and the local features of the two images respectively, and inputting the spliced global features and local features into a decoder to reconstruct a fusion image; and S6: inputting visible light images and SAR images to be fused into the fusion model trained in the above steps to obtain a fused image. The application introduces a Transformer-CNN feature extractor with a double-branch residual structure for image fusion, and improves image processing performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a fusion method of visible light images and SAR images. BACKGROUND

[0002] In recent years, with the continuous development of remote sensing technology, synthetic aperture radar (SAR) imaging as a new imaging technology has become a popular research field. Synthetic aperture radar is an active microwave remote sensing system with characteristics of all-weather, all-day and certain penetration, which can provide SAR images with rich structural information. Compared with traditional visible light images, visible light sensors are susceptible to weather and other factors. Therefore, the organic fusion of visible light and SAR images can greatly enrich the useful information of images, and has important significance in image processing work such as military detection, agricultural planning and target extraction.

[0003] At present, the mainstream visible light and SAR image fusion methods can be roughly divided into two categories, including traditional image fusion methods and deep learning-based image fusion methods. Traditional image fusion algorithms mainly include Laplacian pyramid (LP), shear wave (SW) based image fusion method, wavelet-based image fusion (DWT), non-subsampled shearlet transform (NSST) and sparse representation (SR) methods. However, the traditional methods limit the real-time performance and semantic information integration of the algorithm by using complex transformation and manual rules, which limits the application in advanced visual tasks. Deep learning-based image fusion methods include auto-encoder (AE) framework, convolutional neural network (CNN) framework and generative adversarial network (GAN) framework, which can automatically and efficiently learn the feature information of visible light and SAR images and obtain very accurate fusion results.

[0004] In deep learning-based image fusion methods, autoencoder is a commonly used fusion model, which is mainly composed of an encoder, a fusion decision and a decoder. The encoder is mainly used to encode the source image into a low-dimensional representation in the latent space to obtain the key features of the image. The decoder reconstructs the original image by receiving the latent representation generated by the encoder. During the training process, a suitable loss function is designed to enable the decoder to reconstruct the input image as accurately as possible. After training, the encoder can encode data from different modalities into a low-dimensional representation in the latent space, and then fuse the encoded data according to the designed fusion method, and input the fused encoding into the decoder for reconstruction. The image fusion method based on autoencoder does not need to manually design feature extraction, can effectively learn the key information from the data, and can realize fusion in an end-to-end framework, greatly simplifying the fusion process.

[0005] Among the many AE fusion frameworks, the autoencoder method based on CNN feature extraction and reconstruction is one of the most effective ways. The algorithm based on shared encoder and the method based on private encoder are currently the most commonly used ways of this method. However, there are some problems and deficiencies in these methods: first, CNN is a neural network based on convolutional architecture, which has the characteristics of induction bias and translation invariance. It improves the efficiency of feature calculation, but causes the lack of receptive field, which makes it difficult to extract global information to obtain high-quality fusion images. Second, the forward propagation of the fusion network may cause some important feature information to be lost. Finally, the method based on shared encoder cannot distinguish the features of different modalities, while the method based on private encoder ignores the shared features. They cannot fully consider the different information between different modal images.

[0006] Unlike CNN, the Vision Transformer (ViT) model architecture, which is currently a hot research topic in computer vision, uses self-attention mechanisms, multi-head attention mechanisms, and position encoding to enable the model to effectively capture global dependencies in the input sequence, thereby having excellent global feature extraction capabilities. However, the network model based on ViT is relatively complex and requires a large amount of computation to achieve good performance.

[0007] Therefore, researchers hope to design a better fusion network architecture to solve the deficiencies and challenges faced in feature extraction and fusion of visible light images and SAR images. SUMMARY

[0008] In order to solve the above problems, the application provides a visible light image and SAR image fusion algorithm based on a Transformer and a CNN, which can effectively realize extraction and fusion of global features and local features of the visible light image and the SAR image, so as to improve the subsequent processing performance of the image, and a high-quality visible light and SAR fusion image can be obtained through the algorithm.

[0009] To achieve the above object, the application discloses a visible light image and SAR image fusion method, comprising the following steps:

[0010] S1: a fusion network is constructed based on a Transformer network with an introduced LT model and Dropkey mechanism and a CNN network with an introduced CBAM module;

[0011] S2: visible light images and SAR images of a training set are input into an encoder to respectively extract corresponding global features and local features;

[0012] S3: global features of the visible light images and global features of the SAR images are fused and spliced, the spliced global features are input into a decoder together with local features of the visible light images and local features of the SAR images, and an original image is reconstructed;

[0013] S4: visible light images and SAR images of a training set are input into the encoder trained in step S3 to respectively extract corresponding global features and local features;

[0014] S5: global features of the visible light images and global features of the SAR images are fused and spliced, local features of the visible light images and local features of the SAR images are fused and spliced, and the spliced global features and the spliced local features are input into a decoder to reconstruct a fusion image;

[0015] S6: a visible light image and a SAR image to be fused are input into the fusion model trained in the above steps to obtain a fused image.

[0016] Further, in steps S2 and S4, the feature extraction process comprises: first extracting shallow features of the visible light images and the SAR images by a RestormerBlock, then further extracting global features of the visible light images and the SAR images based on the Transformer network with the introduced LT model and Dropkey mechanism on the basis of the extracted shallow features, and further extracting local features of the visible light images and the SAR images based on the CNN network with the introduced CBAM module.

[0017] Further, step S3 is the first stage of model training, and the total loss calculation formula in this stage is as follows:

[0018] L total_1 = α1L MI + α2L SSIM + α3L decomp (1)

[0019] wherein L total_1 represents the total loss of the first stage image, L MI represents the mutual information loss of the image, L SSIM represents the structural similarity loss, L decomp represents the feature decomposition loss of the image, and α1, α2 and α3 are all adjustment coefficients.

[0020] Further, the adjustment coefficient α1 is 3, the adjustment coefficient α2 is 10, and the adjustment coefficient α3 is 1.

[0021] Further, the calculation formulas of L MI , L SSIM and L decomp are as follows:

[0022] L MI (x, y) = H(x) + H(y) - H(x, y) (2)

[0023] wherein H(x) represents the information entropy of the original image, H(y) represents the information entropy of the reconstructed image, and H(x, y) represents the joint information entropy of the source image and the reconstructed image.

[0024] L SSIM = 1 - SSIM(I VIS' , I VIS ) + β(1 - SSIM(I SAR' , I SAR )) (3)

[0025]

[0026] wherein I VIS′ represents the intensity of the reconstructed visible light image, I SAR′ represents the intensity of the reconstructed SAR image, β is an adjustment coefficient, μ x represents the mean of the original image, μ y represents the mean of the reconstructed image, represents the variance of the original image, represents the variance of the reconstructed image, σ xy represents the covariance of the original image and the reconstructed image, and C1 and C2 are both constants for maintaining stability.

[0027]

[0028] wherein CC(,) represents a correlation coefficient operator, denote local features extracted from the SAR image, denote local features extracted from the visible light image, denote global features extracted from the SAR image, denote global features extracted from the visible light image.

[0029] Further, the calculation formula of C1 and C2 is as follows:

[0030] C1 = (k1L) 2 (7)

[0031] C2 = (k2L) 2 (8)

[0032] wherein, L is the dynamic range of image pixel value, k1 and k2 are both adjustment coefficients.

[0033] Further, the adjustment coefficient β is 0.5, the adjustment coefficient k1 is 0.01, and the adjustment coefficient k2 is 0.03.

[0034] Further, the step S5 is the second stage of model training, in which the total loss calculation formula of training is as follows:

[0035] L total_2 = α4L int + α5L grad + α6L MI + α7L SSIM + α8L decomp (8)

[0036] wherein, L total_2 denotes the total loss of the second stage image, L int denotes the intensity loss of the image, L grad denotes the gradient loss of the image, α4, α5, α6, α7 and α8 are all adjustment coefficients.

[0037] Further, the calculation formula of L int and L grad is as follows:

[0038]

[0039]

[0040] wherein, I SAR denotes the intensity of the reconstructed SAR image, I VIS denotes the intensity of the reconstructed visible light image, I f denotes the intensity of the fusion image, denotes the gradient of the reconstructed SAR image, denotes the gradient of the reconstructed visible light image, denotes the gradient of the fused image, H denotes the height of the image, and W denotes the width of the image.

[0041] Further, the adjustment coefficient a4 is 1, the adjustment coefficient a5 is 1, the adjustment coefficient a6 is 3, the adjustment coefficient a7 is 10, and the adjustment coefficient a8 is 1.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] 1. The present application proposes a dual-branch Transformer-CNN framework to extract and fuse the global high-frequency features and the local low-frequency features of the visible light and SAR data, thereby improving the problem of insufficient feature extraction in the traditional auto-encoder image fusion method.

[0044] 2. Macroscopically, the present application innovatively improves the structure of the dual-branch, that is, instead of separately splicing the global features and the local features of different modal data and then inputting the decoder to reconstruct the original image, the present application splices the global features first, then inputs the spliced global features and the local features of the visible light image and the SAR image into the decoder to reconstruct the original image. In addition, the present application introduces a residual structure into the model, thereby improving the network performance and expression ability and strengthening the extraction ability of complex features in the image.

[0045] 3. Microscopically, the present application innovatively improves the feature extraction network model of the dual-branch, thereby improving the capture ability of important feature information of the network. On the one hand, the LT (LiteTransformer) model is introduced into the Transformer feature extraction network to balance the fused image quality and reduce the calculation cost, and the DropKey mechanism is introduced into the attention layer of the network to adaptively adjust the attention weight, so that the model pays more attention to useful information. On the other hand, the CBAM (Convolutional Block Attention Module) module is introduced into the CNN feature extraction network, the channel attention and spatial attention mechanisms are introduced to improve the focusing ability of the network model on important regions, thereby reducing the loss of important feature information in the forward propagation process of the fusion network and improving the robustness of the model.

[0046] 4. The model training process of the present application is designed in two stages, and the tasks and functions realized in the first stage and the second stage are not completely the same. Therefore, the present application designs a loss function with different characteristics for different training stages to achieve better training effect. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1A structural diagram of a visible light image and SAR image fusion method provided by the present application; wherein: (a) is a flowchart of the first stage of model training; (b) is a flowchart of the second stage of model training; (c) is a principle diagram of the Dropkey mechanism; (d) is a principle diagram of the CBAM module.

[0048] Figure 2 Three pairs of visible light and SAR image pairs with relatively rich texture details selected as contrast display in the embodiments of the present application; from top to bottom are visible light images and SAR images.

[0049] Figure 3 The fusion images obtained after processing the visible light and SAR image pairs in Figure 2 by different image fusion methods in the embodiments of the present application; from top to bottom are the processing results of the DenseFuse method, the processing results of the SeAFusion method, the processing results of the RFN-Nest method, the processing results of the SwinFusion method, the processing results of the YDTR method and the processing results of the method of the present application.

[0050] Figure 4 The radar chart after normalization processing of the quantitative evaluation index of the fusion results in Figure 3 . DETAILED DESCRIPTION

[0051] The present application will be further described below with reference to the accompanying drawings, but in no way limits the present application, any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0052] Referring to Figure 1 , the present application discloses a visible light image and SAR image fusion method, comprising the following steps:

[0053] S1: constructing a fusion network based on a Transformer network introducing an LT model and a Dropkey mechanism and a CNN network introducing a CBAM module;

[0054] S2: inputting the visible light image and the SAR image of the training set into the encoder to extract the respective corresponding global features and local features;

[0055] S3: fusing and splicing the global features of the visible light image and the global features of the SAR image, inputting the spliced global features into the decoder with the local features of the visible light image and the local features of the SAR image respectively to reconstruct the original image;

[0056] S4: inputting the visible light image and the SAR image of the training set into the encoder trained in step S3 to extract the respective corresponding global features and local features;

[0057] S5: Fuse and splice the global features of the visible light image and the global features of the SAR image, fuse and splice the local features of the visible light image and the local features of the SAR image, input the fused global features and the fused local features into a decoder, and reconstruct a fused image;

[0058] S6: input the visible light image and the SAR image to be fused into the fusion model trained in the above steps to obtain a fused image.

[0059] In the steps S2 and S4, the feature extraction process includes: first extracting shallow features of the visible light image and the SAR image by the Restormer Block, then further extracting global features of the visible light image and the SAR image based on the Transformer network introducing the LT model and the Dropkey mechanism on the basis of the extracted shallow features, and further extracting local features of the visible light image and the SAR image based on the CNN network introducing the CBAM module.

[0060] The Restormer Block can extract shallow features of the image without increasing computing power, can promote multi-scale global / local expression learning, and is suitable for image reconstruction tasks. The LT model pays more attention to global information of the image by adopting long-short distance attention, and reduces model parameters by flattening the feedforward network structure, greatly reducing the computing cost while keeping the same performance; at the same time, by introducing the Dropkey mechanism in the attention layer, some key values are randomly discarded, reducing the over-reliance of the model on some neurons, which helps to capture more robust global features. The CBAM module is introduced in the front end of the CNN feature extraction network, which adjusts the importance of different channel information and evaluates the relevance of different spatial positions to improve the focusing ability of the network on important areas, and then obtain more detailed feature information, reducing the loss of important information in the fusion process.

[0061] The step S3 is the first stage of model training, in which the total loss calculation formula of the training is as follows:

[0062] L total_1 =α1L MI +α2L SSIM +α3L decomp (1)

[0063] Wherein, L total_1 represents the total loss of the first stage image, L MI represents the mutual information loss of the image, L SSIM represents the structural similarity loss, and L decompdenotes the feature decomposition loss of the image, and a1, a2 and a3 are all adjustment coefficients and are respectively taken as 3, 10 and 1.

[0064] L MI and L SSIM is designed to reduce the loss of information in the image in the encoding and decoding process, L decomp is a loss function designed to better distinguish the extracted global feature information and local feature information. Their calculation formulas are as follows:

[0065] L MI (x, y) = H(x) + H(y) - H(x, y) (2)

[0066] wherein H(x) denotes the information entropy of the original image, H(y) denotes the information entropy of the reconstructed image, and H(x, y) denotes the joint information entropy of the source image and the reconstructed image;

[0067] L SSIM = 1 - SSIM(I VIS' , I VIS ) + β(1 - SSIM(I SAR' , I SAR )) (3)

[0068]

[0069] wherein I VIS′ denotes the intensity of the reconstructed visible light image, I SAR′ denotes the intensity of the reconstructed SAR image, β is an adjustment coefficient and is taken as 0.5, μ x denotes the mean of the original image, μ y denotes the mean of the reconstructed image, denotes the variance of the original image, denotes the variance of the reconstructed image, σ xy denotes the covariance of the original image and the reconstructed image, and C1 and C2 are both constants for maintaining stability;

[0070]

[0071] wherein CC(,) denotes a correlation coefficient operator, denotes the local feature extracted from the SAR image, denotes the local feature extracted from the visible light image, denotes the global feature extracted from the SAR image, denotes the global feature extracted from the visible light image. Since the visible light image and the SAR image are highly correlated in global feature information, in order to preserve the same global feature of the two types of images, The greater the better, and there is a certain difference between the two types of images in local detail feature information, so as to extract more abundant details, The smaller the better, thus presenting the expression form of formula (5).

[0072] The calculation formula of C1 and C2 is as follows:

[0073] C1=(k1L) 2 (7)

[0074] C2=(k2L) 2 (8)

[0075] Wherein, L is the dynamic range of image pixel value, k1 and k2 are both adjustment coefficients and take 0.01 and 0.03.

[0076] The step S5 is the second stage of model training, in which the total loss calculation formula of training is as follows:

[0077] L total_2 =α4L int +α5L grad +α6L MI +α7L SSIM +α8L decomp (8)

[0078] Wherein, L total_2 indicates the total loss of the second stage image, and L int , L grad two items are added on the basis of the first stage loss function, L int indicates the intensity loss of the image, which constrains the fused image to keep similar intensity distribution with the source image, L grad indicates the gradient loss of the image, which forces the fused image to contain rich texture detail information, and α4, α5, α6, α7 and α8 are all adjustment coefficients and take 1, 1, 3, 10 and 1 respectively.

[0079] The calculation formula of L int and L grad is as follows:

[0080]

[0081]

[0082] Wherein, I SAR indicates the intensity of the reconstructed SAR image, I VIS indicates the intensity of the reconstructed visible light image, I f indicates the intensity of the fused image, indicates the gradient of the reconstructed SAR image, denotes the gradient of the reconstructed visible light image, denotes the gradient of the fused image, H denotes the height of the image, and W denotes the width of the image.

[0083] Next, the advancement and superiority of the above method compared with the existing visible light image and SAR image fusion methods will be demonstrated through specific embodiments.

[0084] 1. Introduction of dataset

[0085] The dataset used in this embodiment is the public dataset OGSOD-1.0 downloaded from the network, which contains 18331 strictly paired visible light and SAR image pairs, wherein the training set contains 14665 visible light and SAR image pairs, and the test set contains 3666 visible light and SAR image pairs. This embodiment has screened the dataset, selected 1048 visible light and SAR image pairs as the training set, and selected 100 visible light and SAR image pairs as the test set.

[0086] 2. Evaluation metrics

[0087] This embodiment quantitatively evaluates the fusion results from 12 commonly used indicators in four aspects of information, structural similarity, image features and human visual perception, wherein the information-based image fusion indicators include entropy (EN), mutual information (MI) and peak signal-to-noise ratio (PSNR); the structural similarity-based image fusion indicators include structural similarity index measurement (SSIM) and mean square error (MSE); the image feature-based image fusion indicators include average gradient (AG), edge intensity (EI), standard deviation (SD), spatial frequency (SF) and edge information-based indicator (Q abf ); the visual perception-based image fusion indicators include sum of correlation differences (SCD) and visual information fidelity (VIF). For the above indicators, except that the smaller the MSE value represents the higher the quality of the fused image, the rest of the indicators are the larger the value, the higher the quality of the fused image.

[0088] 3. Comparison with SOTA methods

[0089] In order to verify the effectiveness of the image fusion method proposed in the present application, the embodiment will compare the fusion results with the currently advanced general image fusion methods, including 5 methods of DenseFuse, RFN-Nest, SeAFusion, SwinFusion and YDTR.

[0090] In order to better compare the fusion performance of different algorithms, this embodiment selects 3 pairs of visible light and SAR image pairs with rich texture details from the test set as a comparison display, as shown inFigure 2 It can be seen from the figure that the visual effect of the visible light image is better, and the local features such as buildings are more clearly expressed, but the contour information is difficult to distinguish from the background; on the contrary, the SAR image is more sufficient in expressing the contour information, so the fused image should contain both local information such as buildings and global information such as terrain contours.

[0091] Figure 3 The intuitive comparison of the fused images obtained by the image fusion method of the present application and the above-mentioned five image fusion methods is shown. It can be seen from the comparison results that the fused image obtained by the method of the present application can obtain more abundant texture details compared with the other five methods, the contour information is also more clear, and the target object in the fused image obtained by the method of the present application is more prominent and is more easily distinguished from the background, which is helpful for better understanding of various scenes.

[0092] In order to more objectively verify the superiority of the method of the present application, Table 1 shows the quantitative index comparison of the method of the present application and the other five methods, wherein the bold data is the optimal value in each index. In order to more intuitively show the comparison effect, the data in Table 1 is normalized and plotted as a radar chart as shown in Figure 4 It can be seen that the method of the present application is the optimal value in the 10 indexes, even if the two indexes (PSNR, MSE) are not the optimal value, they also perform well, which is sufficient to prove that the method of the present application has better performance for the fusion of visible light images and SAR images. The method of the present application performs best in EN, MI indexes compared with the other five methods, which shows that the method of the present application can fully exploit the information of the source images and transform it into the fused image; it performs best in SSIM index, which shows that the method of the present application can fully retain the detail information of the source images and is most similar to the source images; it performs best in AG, EI, SD, SF and Q abf indexes, which shows that the fused image obtained by the method of the present application has better quality and higher clarity; it performs best in SCD, VIF indexes, which shows that the fused image obtained by the method of the present application has better visual effect.

[0093] Table 1 Comparison of result evaluation indexes of different fusion methods

[0094]

[0095]

[0096] 4. Ablation experiments

[0097] The following sets up ablation experiments to verify the rationality of different modules in the method of the application. Mainly, the double branch structure, residual structure Dropkey mechanism, CBAM module and two-stage training used in the experiment are ablation studied, and the specific details are as follows:

[0098] 1. Double branch structure. The application enhances the network's ability to extract global features by introducing a Transformer branch with Dropkey mechanism. In the ablation study, the Transformer branch is replaced by a CNN network branch with CBAM module, and the comparison results are shown in Table 2.

[0099] Table 2 Comparison of fusion indicators with and without double branch structure

[0100]

[0101] 2. Residual structure. By comparing the introduction of residual structure and the non-introduction of residual structure, the comparison results are shown in Table 3.

[0102] Table 3 Comparison of fusion indicators with and without residual structure

[0103]

[0104]

[0105] 3. Dropkey mechanism. For the Transformer branch, comparison experiments are conducted by using Dropkey mechanism and not using Dropkey mechanism, and the comparison results are shown in Table 4.

[0106] Table 4 Comparison of fusion indicators with and without Dropkey mechanism

[0107]

[0108] 4. CBAM module. For the CNN branch, experiments are conducted by using CBAM module and not using CBAM module, and the comparison results are shown in Table 5.

[0109] Table 5 Comparison of fusion indicators with and without CBAM module

[0110]

[0111] 6. Two-stage training. The two-stage training is introduced in this experiment to improve the fusion performance. In the ablation study, the one-stage training method of directly training the encoder, fusion network and decoder is used, and the number of training rounds is consistent with the total number of two-stage training rounds. The comparison results are shown in Table 6.

[0112] Table 6 Fusion index comparison of two-stage training and one-stage training

[0113]

[0114] From the above data comparison, it can be seen that the fusion results of the method of the present application will generally have better indicators, which is sufficient to prove that the image fusion method provided by the present application is reasonable and superior in structure. In addition, it is worth noting that the experimental result indicators obtained by using the Transformer-CNN double branch of the present application are all and significantly higher than the experimental result indicators obtained by using a single CNN branch, and combined with the image comparison of the ablation experiment, it is found that the fusion image obtained by using a single CNN branch has more noise, which is not conducive to the subsequent processing of the image, and the fusion image obtained without using the residual structure has a certain loss of texture details; the fusion image obtained by only one-stage training also has a certain loss of texture details, which all show that the double-branch residual structure is necessary for the method of the present application.

[0115] In addition, in the two groups of comparative experiments of Dropkey and CBAM, the method of the present application has better improvement in the three indicators of PSNR, MSE and SSIM, and combined with the image comparison of the ablation experiment, it is found that the fusion image obtained without using Dropkey and CBAM has obvious loss of detail texture, and the image is relatively blurred, which shows that Dropkey and CBAM used in the method of the present application play a more significant role, so that the obtained fusion image retains more original image information, and has better manifestation in detail texture and other characteristics.

[0116] Therefore, it can be determined that the present application proposes a visible light image and SAR image fusion method based on the double-branch residual structure of the Transformer and CNN network, introduces the LT model and Dropkey mechanism in the feature extraction network based on the Transformer for better extraction of the global features of the two modalities, introduces the CBAM module in the feature extraction network based on the CNN for better extraction of the local features of the two modalities. The present application optimizes and improves the entire fusion network architecture, adds a training stage of first fusing and splicing the global features of the two modalities and then respectively reconstructing with the local features of the respective modalities, and designs a specific loss function to adapt to the task, and finally proves the effectiveness and feasibility of the method of the present application through the embodiments.

[0117] Although the present disclosure has been shown and described with respect to one implementation, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and the annexed drawings. The disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular regard to the various functions performed by the above described components (e.g., elements, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the present disclosure. In addition, while a particular feature of the disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "including", "includes", "having", "has", "contain", "contains", or variants thereof to be afforded similar force or effect to the term "comprising", the terms are intended to encompass the possibilities of having zero or more of the referenced elements present.

[0118] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can exist physically, or a plurality of or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software functional module. If the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0119] In summary, the above-mentioned embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above-mentioned embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principle of the present application are equivalent replacement methods and are included in the protection scope of the present application.

Claims

1. A method of fusing a visible light image and a SAR image, characterized by, Comprising the following steps: S1: constructing a fusion network based on a Transformer network introducing an LT model and a Dropkey mechanism and a CNN network introducing a CBAM module; S2: inputting the visible light image and the SAR image of the training set into the encoder to extract the respective corresponding global features and local features; S3: fusing and splicing the global features of the visible light image and the global features of the SAR image, inputting the spliced global features into the decoder together with the local features of the visible light image and the local features of the SAR image to reconstruct the original image; S4: inputting the visible light image and the SAR image of the training set into the encoder trained in step S3 to extract the respective corresponding global features and local features; S5: fusing and splicing the global features of the visible light image and the global features of the SAR image, fusing and splicing the local features of the visible light image and the local features of the SAR image, and inputting the spliced global features and the spliced local features into the decoder to reconstruct the fused image; S6: inputting the visible light image and the SAR image to be fused into the fusion model obtained by training the above steps to obtain the fused image; In the steps S2 and S4, the feature extraction process comprises: firstly extracting the shallow features of the visible light image and the SAR image by the Restormer Block, and then further extracting the global features of the visible light image and the SAR image based on the Transformer network introducing the LT model and the Dropkey mechanism on the basis of the extracted shallow features, and further extracting the local features of the visible light image and the SAR image based on the CNN network introducing the CBAM module.

2. The method of fusing a visible light image and a SAR image according to claim 1, wherein, The step S3 is the first stage of model training, and the total loss calculation formula in this stage is as follows: （1） wherein, represents a total loss of the first stage image, represents a mutual information loss of the image, represents a structural similarity loss, represents a feature decomposition loss of the image, , and are adjustment coefficients.

3. The method of fusing a visible light image and a SAR image according to claim 2, wherein, adjustment coefficient is 3, adjustment coefficient is 10, adjustment coefficient is 1. 4.The method of fusing a visible light image and a SAR image according to claim 2, characterized in that, , and The calculation formula is as follows: （2） wherein, denotes the information entropy of the original image, denotes the information entropy of the reconstructed image, denotes the joint information entropy of the source image and the reconstructed image; （3） （4） wherein, denotes the intensity of the reconstructed visible light image, denotes the intensity of the reconstructed SAR image, is an adjustment factor, denotes the mean of the original image, denotes the mean of the reconstructed image, denotes the variance of the original image, denotes the variance of the reconstructed image, denotes the covariance of the original image and the reconstructed image, and are both constants for maintaining stability. （5） where CC(, ) denotes a correlation coefficient operator, denotes local features extracted from the SAR image, denotes local features extracted from the optical image, denotes global features extracted from the SAR image, denotes global features extracted from the optical image.

5. The method of fusing a visible light image and a SAR image according to claim 4, wherein, and The calculation formula is as follows: （7） （8） wherein, L is a dynamic range of image pixel values, and are adjustment coefficients.

6. The method of fusing a visible light image and a SAR image according to claim 5, wherein, adjustment coefficient is 0.5, adjustment coefficient is 0.01, adjustment coefficient is 0.

03.

7. The method of fusing a visible light image and a SAR image according to claim 5, wherein, The step S5 is the second stage of model training, and the total loss calculation formula in this stage is as follows: （8） wherein, represents the total loss of the second stage image, represents the intensity loss of the image, represents the gradient loss of the image, , , , and are adjustment coefficients.

8. The method of fusing a visible light image and a SAR image according to claim 7, wherein, and The calculation formula is as follows: （9） （10） wherein, denotes the intensity of the reconstructed SAR image, denotes the intensity of the reconstructed visible light image, denotes the intensity of the fused image, denotes the gradient of the reconstructed SAR image, denotes the gradient of the reconstructed visible light image, denotes the gradient of the fused image, H denotes the height of the image, and W denotes the width of the image.

9. The method of fusing a visible light image and a SAR image according to claim 8, wherein, adjustment coefficient is 1, adjustment coefficient is 1, adjustment coefficient is 3, adjustment coefficient is 10, adjustment coefficient is 1.

Images