SAR-optical image fusion method based on hybrid model

Abstract

The present invention discloses a SAR-optical image fusion method based on a hybrid model, comprising: acquiring a plurality of SAR images and optical images; preprocessing the SAR and optical images to obtain a first image pair; acquiring remote sensing image data; adding noise to the remote sensing image data to obtain a first dataset; end-to-end training a multiscale global noise suppression network based on the first dataset to obtain a first model; inputting the SAR training image into the first model to obtain a first input image as output; obtaining a second input image based on the grayscale channels of the optical training image; combining the first and second input images to form a second image pair to form a second dataset; and constructing a multiscale global fusion model to obtain a trained second model. By achieving pixel-level fusion of SAR and optical images, it is possible to avoid the problem of unclear feature extraction and information loss in the fusion result, thereby ensuring that the information contained in the resulting fused image is more comprehensive and accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to a SAR-optical image fusion method based on a hybrid model. Background Art

[0002] Synthetic aperture radar (SAR) is a state-of-the-art technology that actively transmits microwaves to receive backscattered energy from ground objects at any time and anywhere. Different backscattered energies from the same object vary depending on its roughness, moisture content, and dielectric properties, enabling the acquisition of distinct information about different ground objects. Due to its long wavelength, its radiation has strong penetrating power, capable of penetrating all climatic conditions except clouds, haze, dust, and heavy rainfall. This characteristic contributes to the all-weather, all-day advantage of SAR imagery. SAR images primarily depict the structural features of various objects on the image surface. SAR can identify different objects within an image based on their surface features, resulting in SAR images with richer spatial information.

[0003] Existing optical sensors are passive sensors that receive a portion of visible light reflected from Earth objects. These sensors cover the ultraviolet, visible, and infrared portions of the electromagnetic spectrum. Optical sensors can be categorized as panchromatic, multispectral, and hyperspectral sensors. Panchromatic sensors are sensitive to a wide range of wavelengths, encompassing the visible and near-infrared portions of the electromagnetic spectrum. Visible light images are rich in ground texture details, complementing details lost in SAR images during the imaging process.

[0004] Clearly, SAR images produced by existing technologies are subject to the influence of smear noise, resulting in poor interpretability. Optical images rely on sunlight reflected from Earth objects. Two objects with different structures may appear identical in optical imaging, but their spectral responses are distinguishable in SAR imaging. Therefore, SAR and optical imaging provide complementary information about the imaged area, and the fusion of these images helps to generate images rich in spatial and spectral information. In addition, existing image fusion methods are mostly based on convolutional fusion, which cannot guarantee the capture of global image features. To ensure more comprehensive acquisition of useful information from both types of images during image fusion while improving resolution, a hybrid model-based SAR-optical image fusion method is urgently needed. This method can achieve multi-scale image fusion by adding a Transformer structure to a CNN architecture. The Transformer is a neural network model based on an attention mechanism. The Transformer is an "encoder-decoder" architecture consisting of an encoder and a decoder, both of which are superimposed with multi-head self-attention modules.

[0005] Patent Document 1 (Publication Number: CN107480701A) discloses a method for matching optical and radar images based on a multi-channel convolutional neural network. The method comprises: 1) extracting features from the optical image using a deep convolutional neural network to obtain 32×32×32-dimensional image feature data for the optical image; 2) extracting features from the SAR image using another deep convolutional neural network to obtain 32×32×32-dimensional image feature data for the SAR image; 3) concatenating the image feature data extracted from the SAR and optical images to form a joint feature; and 4) constructing a matching network based on the joint feature, performing fully connected matching classification, and outputting the matching results. Although this method matches optical and radar images, it does not incorporate a Transformer structure for multi-scale image fusion, and therefore cannot guarantee the capture of global image features.

[0006] Patent Document 2 (Publication No.: CN112634185A) discloses a method for fusing SAR images and optical images based on HSL and image entropy. The method comprises the following steps: S1, reading in the SAR image and optical image to be fused, first transforming the optical image to be fused from the RBG model to the HSL space, and then processing it in the HSL space. The HSL transformation can effectively separate the image brightness L, chroma H, and saturation S in the RGB model to obtain the chroma component H, the saturation component S, and the brightness component L; S2, performing contrast expansion on the chroma component H and the saturation component S, respectively, to obtain the chroma component H1 and the saturation component S1. Although this method fuses optical images with radar images, it does not add a Transformer structure to extract global features and perform multi-scale image fusion, and cannot guarantee the capture of global and deep features of the image. Summary of the Invention

[0007] In view of this, the present invention provides a SAR-optical image fusion method based on a hybrid model.

[0008] This application provides a SAR-optical image fusion method based on a hybrid model, comprising:

[0009] Acquire several SAR images and optical images of the same area;

[0010] Preprocessing the SAR image and the optical image to obtain a first image pair; the first image pair includes: a SAR training image and a corresponding optical training image;

[0011] Acquisition of remote sensing image data;

[0012] The remote sensing image data is added with noise to simulate the SAR training image to obtain a first data set; wherein the surrogate model of the noise is expressed as follows:

[0013] J = ηI;

[0014] J=I+(η-1)I;

[0015] J = I + K;

[0016] Where J is the observed image intensity, I is the noise-free image, K = (η-1)I is the multiplicative speckle component of the SAR image, η is a random sampling with mean μ and variance σ∈[0.55,0.9];

[0017] Performing end-to-end training on a multi-scale global noise suppression network according to the first data set until the training is terminated to obtain a first model;

[0018] Inputting at least part of the SAR training images into the first model in batches to obtain a first input image as output; the first input image is a clean SAR image after denoising;

[0019] Obtaining a second input image based on at least a portion of the grayscale channels of the optical training image;

[0020] Cropping the first input image and the second input image to the same size, and forming a second image pair with images having the same content, thereby forming a second data set;

[0021] Sequentially constructing an image feature extraction structure, a feature fusion structure, and a feature reconstruction structure; connecting the image feature extraction structure, the feature fusion structure, and the feature reconstruction structure to form a multi-scale global fusion model of SAR-optical images, wherein the multi-scale global fusion model is a second model;

[0022] According to the structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse The second model is trained until the objective function L fuse It converges to the minimum value and does not decrease any more, and the second model after training is obtained.

[0023] Optionally, preprocessing the SAR image and the optical image to obtain a first image pair includes:

[0024] Optical registration and cropping are performed on the SAR image and the optical image to form a first image pair with the same size and matching content.

[0025] Optionally, simulating the SAR image by adding noise to the remote sensing image data to obtain a first data set includes:

[0026] A coherent speckle noise modeling function is obtained, and the coherent speckle noise is modeled as a noise in a product form, which is expressed as follows:

[0027] y=x·n; (1)

[0028] Where x is a clean grayscale image, n is the modeled noise, and y is the synthesized noise image;

[0029] The SAR image is a single-view image, and n obeys a Gamma distribution with unit mean and variance 1 / L. The probability distribution of the modeling noise n is:

[0030]

[0031] Wherein, P(n) is the probability distribution of the modeling noise n, Γ(n) is the Gamma function, L is the number of observations, n is the modeling noise, the variance is 1 / L, and e is the base of the natural logarithm;

[0032] The remote sensing image data is used as the clean grayscale image x, the clean grayscale image x is processed, the modeling noise n is added, and the noise image y is synthesized to form the first data set of the synthesized noise.

[0033] Optionally, the method for constructing the multi-scale global noise suppression network includes:

[0034] The encoder and decoder are designed based on the U-shaped structure. The encoder is responsible for reducing the size of the image and realizing feature extraction, and the maximum pooling layer is used to realize 2, 4, and 6 times downsampling. After each downsampling, the image size is reduced and the receptive field of the image becomes larger. The Transformer structure is used to extract global features at each size. The decoder is responsible for gradually restoring the feature map of the downsampled image with a resolution lower than the original clarity to the original input size, and then using deconvolution to realize 2, 4, and 6 times upsampling to restore the image with original clarity. After each upsampling, a Transformer structure is also used for feature recovery to restore the denoised image. At the same time, the feature maps of the same size of the encoder and the decoder are connected to obtain the multi-scale global noise suppression network.

[0035] Optionally, performing end-to-end training on a multi-scale global noise suppression network according to the first data set until the training is terminated to obtain a first model includes:

[0036] During the training process, according to the loss function And the total variation loss function Find the linear sum and get the final loss function According to the final loss function Train the first model to obtain the final loss function of the first model

[0037] The loss function for the first model The total variation loss function And the final loss function The expressions are:

[0038]

[0039] in, is the loss function, is the total variational loss function, is the final loss function, λ1 is the loss function The weight of , λ1=1, λ2 is the total variation loss function The weight of λ2=5×10 -5 , x is a clean image, represents the output image of the first model, i represents the i-th row, and j represents the j-th column.

[0040] Optionally, obtaining the second input image based on at least a portion of the grayscale channel of the optical training image includes:

[0041] The grayscale channel of the optical training image is a full-color channel of visible light. A Y channel is obtained according to the full-color channel of visible light, and the second input image is obtained according to the Y channel. The expression of the Y channel is:

[0042] Y=0.257×R+0.564×G+0.098×B+16; (6)

[0043] Among them, R is the red channel of the RGB color image of visible light input, G is the green channel of the RGB color image of visible light input, and B is the blue channel of the RGB color image of visible light input.

[0044] Optionally, the method for constructing the image feature extraction structure includes:

[0045] The convolution structure and the Transformer structure are alternately mixed and connected. The convolution structure is responsible for reducing the image scale, extracting local features, and mapping the input image to a higher dimension. The Transformer structure is responsible for extracting global information, and the weight of the acquired sequence information relative to the global information is calculated according to the attention mechanism. The constructed feature extraction network is used to extract global and local features of the image to obtain the image feature extraction structure.

[0046] Optionally, the method for constructing the feature fusion structure includes:

[0047] Based on a Transformer-based fusion module and a CNN-based fusion module, a dual-branch hybrid fusion structure is designed; wherein, the Transformer fusion module mainly uses a cross-attention mechanism to complete the interaction between features, and the CNN fusion module uses the local weighted computing capability of convolution and the splicing characteristics between channels to complete the interaction of features; the dual-branch hybrid fusion structure is based on a fusion method of global and local features, and finally the fused features of the two branches are spliced ​​according to the channels to obtain a fusion feature map after the fusion of global and local features, thereby obtaining the feature fusion structure.

[0048] Optionally, the method for constructing the feature reconstruction structure includes:

[0049] The deconvolution structure is used to restore the image that has been reduced during the feature extraction process to the same size as the input size; wherein the feature reconstruction object is the fusion feature map of the SAR image and the optical image through the feature fusion structure, which represents the comprehensive feature representation of the SAR image and optical image information. The fusion feature map is reconstructed and the abstract features are converted back to the original image space.

[0050] Optionally, the structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse The second model is trained until the objective function L fuse Converging to the minimum value and no longer decreasing, the second model after training is obtained, including:

[0051] The objective function L of the second model fuse Satisfies the following expression:

[0052] L feat =L fuse -αL det ; (7)

[0053] Among them, L feat is the feature similarity loss, L fuse is the objective function, ɑ is a constant term, 0<ɑ<1, L det is the structural similarity loss;

[0054] The structural similarity loss L det The calculation formula is:

[0055] Ldet =1-SSIM(O,I); (8)

[0056] Wherein, O is the fusion result image output by the second model of the first input image and the second input image; I is the first input image input by the second model; SSIM (.) is used to measure structural similarity;

[0057] The feature similarity loss L feat The expression is:

[0058]

[0059] Among them, w I , w1 and w2 are the trade-off parameters for balancing the loss size, both are numbers between 0 and 1, Φ f represents the feature map output by the second model; Φ1 is the optical image input by the second model, that is, the first feature map obtained by the second input image through the feature fusion structure of the second model; Φ2 is the second feature map obtained by the second input image through the feature fusion structure of the second model, and F represents the F-norm calculation.

[0060] Compared with the prior art, the hybrid model-based SAR-optical image fusion method provided by the present invention achieves at least the following beneficial effects:

[0061] The SAR-optical image fusion method based on the hybrid model provided by the present invention obtains SAR images and optical images of the same area, performs optical registration on the two types of images, and uses them as training data sets for input deep learning framework; designs a first module for the initial SAR image to improve the signal-to-noise ratio before fusion, models the coherent speckle noise of the SAR image, designs a multi-scale noise suppression network based on deep learning to suppress global noise, uses a synthetic remote sensing image dataset to train the network until the loss converges, and uses the Y channel of the SAR image after image quality enhancement and the visible light image as the image to be fused; designs a feature extraction module, and uses a hybrid structure composed of alternating convolution and Transformer to simultaneously shallow local features and deep semantic features are extracted at the same time; the feature fusion module is divided into a Transformer fusion branch and a convolution fusion branch, and the final fusion features are output after channel superposition; the image reconstruction module reconstructs the extracted fusion features into a final fusion image; through the above clear and concise steps, the pixel-level fusion of SAR images and optical images is realized, and the feature extraction is not obvious, which avoids the problem of missing information in the fusion result, so that the information contained in the obtained fusion image is more comprehensive and accurate, suitable for various occasions, and can also be used to improve the subsequent image recognition, image tracking and other aspects caused by insufficient signal-to-noise ratio and insufficient information of SAR images in the existing technology.

[0062] Of course, any product implementing the present invention does not necessarily need to achieve all of the technical effects described above at the same time.

[0063] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0065] Figure 1 is a flow chart of a SAR-optical image fusion method based on a hybrid model provided by an embodiment of the present invention;

[0066] Figure 2a is a clean image after denoising the SAR image provided by an embodiment of the present invention;

[0067] Figure 2b is a noise image before SAR image denoising provided by an embodiment of the present invention;

[0068] Figure 3a is a SAR image provided by an embodiment of the present invention;

[0069] Figure 3b is a visible light image provided by an embodiment of the present invention;

[0070] Figure 3c This is the final fused image provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0071] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0072] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0073] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0074] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0075] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0076] See also Figure 1 As shown in Figure 3, Figure 1 is a flow chart of a SAR-optical image fusion method based on a hybrid model provided by an embodiment of the present invention; Figure 2a is a clean image after denoising the SAR image provided by an embodiment of the present invention; Figure 2b is a noise image before SAR image denoising provided by an embodiment of the present invention; Figure 3a is a SAR image provided by an embodiment of the present invention; Figure 3b is a visible light image provided by an embodiment of the present invention; Figure 3c is the final fused image provided by an embodiment of the present invention. This embodiment provides a SAR-optical image fusion method based on a hybrid model, specifically a SAR-optical image pixel-level fusion method based on a multi-scale global model, including:

[0077] S1, acquire several SAR images and optical images of the same area;

[0078] Specifically, this embodiment can obtain SAR images and optical images of the same area through the Gaofen-3 satellite and the Google Earth satellite. The obtained optical image should ensure that the information at the time of SAR image acquisition is consistent. The image of the same area can be obtained by filling in the longitude and latitude. Furthermore, the pre-processed SAR image comes from the Gaofen-3 satellite, and the visible light image comes from the virtual globe.

[0079] S2, preprocessing the SAR image and the optical image to obtain a first image pair; the first image pair includes: a SAR training image and a corresponding optical training image;

[0080] Specifically, in step S2, the SAR image and the optical image are preprocessed to obtain a first image pair, including:

[0081] Combine Figure 3a and Figure 3bAs shown, the SAR image and the optical image are optically registered and cropped to form a first image pair of the same size and matching content, thereby obtaining a SAR-optical image pair of 256*256 pixels. While forming a SAR-optical image pair suitable for deep learning, it can also be decomposed into a training set and a test set suitable for deep learning in a certain ratio. In other words, the process includes registering the two types of images, cropping the images into image blocks of 256*256 pixels, and forming the images with the same content into a first image pair of a SAR training image and an optical training image corresponding to the SAR training image. The SAR can be a real SAR training image.

[0082] Optionally, manual registration can be performed during the registration process. This involves manually aligning the SAR and optical images, locating image matching points, and selecting geometrically invariant corner points of buildings, ships, and roads to complete the coordinate mapping between the images. The registration process involves manually selecting corner points and feature points with consistent spatial relative positions in the SAR and optical images, calculating the mapping relationship between the same points in the two images, and applying the resulting mapping relationship to the SAR image to correct it. The resulting corrected image forms a pair of registered images with the optical image.

[0083] S3, acquiring remote sensing image data;

[0084] Specific, combined Figure 2a As shown, it comes from a public remote sensing image dataset; in the dataset, each category contains 100 pictures, totaling 21 land use categories; the remote sensing image data includes original images such as airplanes, buildings and rivers, and synthetic images are generated from the remote sensing images by using prior statistics on speckle; the remote sensing image data can specifically be aerial remote sensing image data.

[0085] S4, using the remote sensing image data plus noise to simulate the SAR image to obtain a first data set; wherein the noise substitution model is expressed as follows:

[0086] J = ηI;

[0087] J=I+(η-1)I;

[0088] J = I + K;

[0089] Where J is the observed image intensity, I is the noise-free image, K = (η-1)I is the multiplicative speckle component of the SAR image, η is a random sampling with mean μ and variance σ∈[0.55,0.9];

[0090] Specific, combined Figure 2b As shown, in step S4, the remote sensing image data is added with noise to simulate the SAR image to obtain a first data set, including:

[0091] Obtain the coherent speckle noise modeling function and model the coherent speckle noise as a product form of noise. The expression is:

[0092] y=x·n; (1)

[0093] Where x is a clean grayscale image, n is the modeled noise, and y is the synthesized noise image;

[0094] Assuming that the SAR image is a single view, n obeys a Gamma distribution with unit mean and variance 1 / L, and the probability distribution of the modeling noise n is:

[0095]

[0096] Where P(n) is the probability distribution of the modeling noise n, Γ(n) is the Gamma function, L is the number of observations, n is the modeling noise, the variance is 1 / L, and e is the base of the natural logarithm;

[0097] The remote sensing image data is used as a clean grayscale image x, the clean grayscale image x is processed, modeling noise n is added, and a noisy image y is synthesized to form a first data set of synthetic noise.

[0098] In this embodiment, the noise can be added manually to the remote sensing image data according to the modeling formula of speckle noise to synthesize the first data set.

[0099] It should be noted that Gamma Distribution is a continuous probability function in statistics and a very important distribution in probability statistics. 2 "Distribution" is a special case of the gamma distribution; the gamma function (Gamma function), also known as Euler's second integral, is a type of function that extends the factorial function on real numbers and complex numbers. This function has important applications in analysis, probability theory, partial differential equations, and combinatorics. The noise model is multiplicative noise; the remote sensing image data can be high-quality remote sensing images, that is, high-resolution remote sensing images; the resolution of the high-resolution remote sensing images is not less than 1m.

[0100] S5, performing end-to-end training on the multi-scale global noise suppression network according to the first data set until the training is terminated to obtain a first model;

[0101] Specifically, the Unet structure, also known as the U-Net structure, is a convolutional neural network for image segmentation. The Unet structure is used to achieve 2, 4, and 6 times downsampling and upsampling, and the original local feature extraction structure (referring to the CNN structure in the Unet network, CNN, or Convolutional Neural Networks, is a type of feedforward neural network with a deep structure that includes convolution calculations and is one of the representative algorithms of deep learning. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input information according to their hierarchical structure) is updated to a global feature extraction structure to obtain a denoising model, which is the first model;

[0102] In step S5, the method for constructing a multi-scale global noise suppression network includes:

[0103] The encoder and decoder are designed based on the U-shaped structure. The encoder is responsible for reducing the size of the image and realizing feature extraction, and the maximum pooling layer is used to realize 2, 4, and 6 times downsampling. After each downsampling, the image size is reduced and the receptive field of the image becomes larger. The Transformer structure is used to extract global features at each size. The decoder is responsible for gradually restoring the feature map of the downsampled image with a resolution lower than the original clarity to the original input size, and then using deconvolution to realize 2, 4, and 6 times upsampling to restore the image with original clarity. After each upsampling, a Transformer structure is also used for feature recovery to restore the denoised image. At the same time, the feature maps of the same size of the encoder and the decoder are connected to obtain the multi-scale global noise suppression network.

[0104] The generation of this low-resolution feature map is due to the fact that the scale gradually decreases and the content gradually becomes abstract during the image downsampling process. The details have been ignored, and only some significant features are left, not pixel-level features, so the image clarity is reduced.

[0105] The encoder and decoder are designed based on the U-shaped structure. The encoder is responsible for reducing the size of the image and realizing feature extraction. The maximum pooling layer is used to achieve 2, 4, and 6 times downsampling. After each downsampling, the image size is reduced and the receptive field of the image becomes larger. Then, the Transformer structure is immediately used to extract global features. The Transformer structure includes image sequence embedding, attention mechanism, and feedforward neural network parts, which can effectively extract the global features of the image. The decoder part is similar to the encoder. It is responsible for gradually restoring the low-resolution feature map after downsampling to the original input size, that is, using deconvolution to achieve 2, 4, and 6 times upsampling. After each upsampling, a Transformer structure is immediately used for decoding to restore the denoised image. At the same time, in order to better utilize the information in the encoder, the feature maps of the same size of the encoder and decoder are connected to obtain a multi-scale global noise suppression network. This can help to better restore the original image;

[0106] In step S5, the multi-scale global noise suppression network is trained end-to-end according to the first data set until the training is terminated to obtain a first model, including:

[0107] During the training process, according to the loss function And the total variation loss function Find the linear sum and get the final loss function According to the final loss function Train the first model and get the final loss function of the first model

[0108] Loss function for the first model Total variation loss function And the final loss function The expressions are:

[0109]

[0110] in, is the loss function, is the total variational loss function, is the final loss function, λ1 is the loss function The weight of , λ1=1, λ2 is the total variation loss function The weight of λ2=5×10 -5 , x is a clean image, represents the output image of the first model, i represents the i-th row, and j represents the j-th column;

[0111] In machine learning algorithms, the loss function measures the difference between predicted values ​​and true values. It's a key component of model optimization, with the goal of minimizing the loss function. Common loss functions include mean squared error (MSE) and cross-entropy. The total variation loss (TV Loss) acts as a regularization term in conjunction with the loss function to regulate network learning. The final loss function is the sum of the TV Loss and the total variation loss, indicating that both loss functions act together on the network.

[0112] S6, inputting at least part of the SAR training images into the first model in batches to obtain a first input image as output; the first input image is a clean SAR image after denoising;

[0113] Specifically, the first model is a speckle suppression module that enhances the image quality of SAR images and improves the signal-to-noise ratio of SAR images with fusion using a multi-scale global denoising network.

[0114] In this embodiment, all SAR training images can be input into the first model to obtain the output first input image; of course, according to actual needs, in this embodiment, part of the SAR training images can also be input into the first model to obtain the output first input image, and this embodiment does not limit this.

[0115] S7, obtaining a second input image based on at least a portion of the grayscale channel of the optical training image;

[0116] Specifically, in step S7, obtaining a second input image according to at least a portion of the grayscale channel of the optical training image includes:

[0117] The grayscale channel of the optical training image is the full-color channel of visible light. The Y channel is obtained according to the full-color channel of visible light, and the second input image is obtained according to the Y channel. The expression of the Y channel is:

[0118] Y=0.257×R+0.564×G+0.098×B+16; (6)

[0119] Among them, R is the red channel of the RGB color image of visible light input, G is the green channel of the RGB color image of visible light input, and B is the blue channel of the RGB color image of visible light input.

[0120] Optionally, in this embodiment, the second input image can also be obtained based on the grayscale channels of all optical training images; of course, according to actual needs, this embodiment can also obtain the second input image based on the grayscale channels of all optical training images, and this embodiment does not specifically limit this.

[0121] S8, cropping the first input image and the second input image to the same size, and forming a second image pair with images having the same content, thereby forming a second data set;

[0122] Specifically, the second image pair may be a SAR-optical image pair. The clean SAR image in the second image pair is the resultant image obtained by applying the first model to the SAR image in the first image pair. The optical image in the second image pair is the resultant image obtained by taking the grayscale channel of the optical image in the first image pair.

[0123] S9, sequentially constructing an image feature extraction structure, a feature fusion structure, and a feature reconstruction structure; connecting the image feature extraction structure, the feature fusion structure, and the feature reconstruction structure to form a multi-scale global fusion model of SAR-optical images, the multi-scale global fusion model being the second model;

[0124] Specifically, the image feature extraction structure is used to obtain the features of the two modalities of SAR images and optical images respectively. The feature extraction methods of SAR images and optical images are the same. The image feature extraction structure is composed of a convolution with downsampling characteristics and a Transformer. Among them, the convolution can perform downsampling and local shallow feature extraction, and the Transformer structure can capture long-range features, thereby improving the multimodal image extraction capability. The extracted image features include: shallow features of the first input image, deep features of the first input image, shallow features of the second input image, and deep features of the second input image.

[0125] The feature fusion structure is used to fuse the complementary information in SAR images and optical images. The dual-stream feature fusion structure is used to fuse the global features and local features of the two modalities and set the loss function. The feature reconstruction network is used to restore the fused feature image to a single fused image.

[0126] The first shallow features include edge information and texture details in the SAR image; the first deep features include shape and object contour information in the SAR image; the second shallow features include edge information and texture details in the optical image; the second deep features include shape and object contour information in the optical image;

[0127] In step S9, the method for constructing the image feature extraction structure includes:

[0128] The convolutional structure and the Transformer structure are alternately mixed and connected. The convolutional structure is responsible for reducing the image scale, extracting local features, and mapping the input image to a higher dimension. The Transformer structure is responsible for extracting global information. The weight of the acquired sequence information relative to the global information is calculated according to the attention mechanism. The constructed feature extraction network is used to extract global and local features of the image to obtain an image feature extraction structure.

[0129] It should be noted that each extraction module in the constructed feature extraction network is composed of a convolution and a Transformer structure. The convolution is responsible for local feature extraction and up and down sampling, and the Transformer structure is responsible for global feature extraction. SAR images and optical images use the same multi-scale feature extraction method. Furthermore, since the input feature dimensions of the convolution are B, C, H, W, where B is the batch size, specifically the number of images input in each batch; C is the number of image channels; H is the image height; W is the image width; L is the number of input patch image blocks; and the input feature dimensions of the Transformer structure are B, L, and C, when the convolution is connected to the Transformer, its dimensions need to be reconstructed. In addition, the Transformer structure does not change the input and output channels, so attention should be paid to the change in dimension when the two are alternately connected.

[0130] In step S9, the method for constructing the feature fusion structure includes:

[0131] Based on a Transformer-based fusion module and a CNN-based fusion module, a dual-branch hybrid fusion structure is designed; wherein, the Transformer fusion module mainly uses a cross-attention mechanism to complete the interaction between features, and the CNN fusion module uses the local weighted computing capability of convolution and the splicing characteristics between channels to complete the interaction of features; the dual-branch hybrid fusion structure is based on a fusion method of global and local features, and finally the fused features of the two branches are spliced ​​according to the channels to obtain a fusion feature map after the fusion of global and local features, thereby obtaining the feature fusion structure.

[0132] It should be noted that this structure utilizes a hybrid fusion method based on global and local features. The dual-branch hybrid fusion structure is a dual-stream fusion network structure, including a Transformer branch and a convolution branch. The Transformer branch is responsible for long-range feature fusion, and the convolution branch is responsible for local feature fusion. The number of channels is spliced ​​to ensure the consistency of the channels before and after fusion. The dual-stream fusion network structure accepts SAR image features and optical image features from the feature extraction network respectively. The Transformer structure branch can accelerate network convergence and use the attention mechanism to obtain global feature weights, enhance the global expression of features, and achieve the purpose of global parameter sharing. The convolution feature locally shares parameters, enhancing the correlation of local features. The dual-stream network enables a more comprehensive fusion of extracted multi-scale features.

[0133] In step S9, the method for constructing the feature reconstruction structure includes:

[0134] The deconvolution structure is used to restore the image that has been reduced during the feature extraction process to the same size as the input size; wherein the feature reconstruction object is the fusion feature map of the SAR image and the optical image through the feature fusion structure, which represents the comprehensive feature representation of the SAR image and optical image information. The fusion feature map is reconstructed and the abstract features are converted back to the original image space.

[0135] It should be noted that the feature reconstruction object is the comprehensive feature representation of the SAR image and the optical image through the feature fusion structure, which has a lower resolution than the input image. In order to restore the lost detail information, it is necessary to convert the fused abstract features back to the original image space; the feature reconstruction structure is the inverse process of the feature extraction structure, specifically referring to swapping the input and output ends of the image feature extraction.

[0136] It should be noted that the process of each input transfer to the output of the second model is specifically as follows: input the second data set into the second model, the second data set including the second image pair composed of the first input image and the second input image after processing; first use the image feature extraction structure to extract image features, the extracted image features include: shallow features of the first input image, deep features of the first input image, shallow features of the second input image and deep features of the second input image; then according to the feature fusion structure, the shallow features of the first input image, the shallow features of the second input image, the deep features of the first input image and the deep features of the second input image are fused respectively to obtain a fused feature map; combined Figure 3c As shown, finally, the fused feature map is reconstructed into a final fused image according to the feature reconstruction structure; the final fused image is the output image of the second model; the fused image represents the required output result, that is, it contains the complementary features of the SAR image and the optical image;

[0137] The feature fusion architecture uses convolution and Transformer to construct feature fusion branches, fusing shallow and deep features of SAR and optical images, respectively. Finally, channel splicing is performed to ensure consistent input and output features. Finally, an inverse upsampling process is designed based on feature extraction rules to produce the final fused image.

[0138] S10, based on the structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse Train the second model until the objective function L fuse It converges to the minimum value and no longer decreases, and the second model after training is obtained.

[0139] Specifically, to obtain a good second model, it is necessary to determine an objective function and optimize the second model so that after each input of the second model is passed to the output, the loss function value calculated between the input and output gradually converges, thereby achieving the purpose of optimizing the model. This optimization method utilizes the second dataset and trains the overall network structure of the second model according to the objective function. The input represents the content of the second dataset, which is passed through the second model to obtain the output. By optimizing the overall network structure of the second model, it becomes a second model that can achieve effective fusion functions.

[0140] Structural similarity loss is structural similarity. Structural similarity (SSIM) indicates the degree of similarity between two images and is a commonly used image quality evaluation indicator. The SSIM calculation formula is as follows:

[0141] SSIM(x,y)=[l(x,y)]^α*[c(x,y)]^β*[s(x,y)]^γ;

[0142] l(x,y) represents brightness similarity, c(x,y) represents contrast similarity, s(x,y) represents structural similarity, and α, β, and γ are constants for adjusting the weights of different components.

[0143] The SSIM value range is between [-1, 1]. A larger value indicates a higher structural similarity between the two images. In this embodiment, the structural similarity is used to calculate the similarity between the fused image output by the second model and the first input SAR image.

[0144] The objective function is the form of the goal pursued expressed by the design variables. The objective function is a function of the design variables and is a scalar.

[0145] Specifically, in step S12, according to the structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse Train the second model until the objective function L fuse Converges to the minimum value and no longer decreases, and obtains the second model after training, including:

[0146] The objective function L of the second model fuse Satisfies the following expression:

[0147] L feat =L fuse -αL det ; (7)

[0148] Among them, L feat is the feature similarity loss, L fuse is the objective function, ɑ is a constant term, 0<ɑ<1, L det is the structural similarity loss;

[0149] The structural similarity loss L det The calculation formula is:

[0150] L det =1-SSIM(O,I); (8)

[0151] Wherein, O is the fusion result image output by the second model of the first input image and the second input image; I is the first input image input by the second model; SSIM (.) is used to measure structural similarity;

[0152] The feature similarity loss L feat The expression is:

[0153]

[0154] Among them, w I , w1 and w2 are the trade-off parameters for balancing the loss size, both are numbers between 0 and 1, Φ f represents the feature map output by the second model; Φ1 is the optical image input by the second model, that is, the first feature map obtained by the second input image through the feature fusion structure of the second model; Φ2 is the second feature map obtained by the second input image through the feature fusion structure of the second model, and F represents the F-norm calculation.

[0155] It should be noted that in expression (7), it is necessary to minimize the loss of the fusion network, retain fine structural details, and retain contour details; in expression (8), O and I are the fused source image and the input source image respectively. SSIM(.) measures structural similarity. If SSIM(O,I) approaches 1, the fused image retains most of the structural details of the source image. feat ; In expression (9), the loss constrains the fused deep features to preserve salient structures, thereby enhancing the fused feature space to learn more salient features and preserve fine details.

[0156] The hybrid model-based SAR-optical image fusion method provided in this embodiment avoids the global method used in the prior art, realizes pixel-level fusion of SAR images and optical images, and can avoid the problem of unclear feature extraction and information loss in the fusion result. The information contained in the obtained fused image is more comprehensive and accurate, and is suitable for a variety of occasions.

[0157] It can be seen from the above embodiments that the SAR-optical image fusion method based on the hybrid model provided by the present invention achieves at least the following beneficial effects:

[0158] The SAR-optical image fusion method based on the hybrid model provided by the present invention obtains SAR images and optical images of the same area, performs optical registration on the two types of images, and uses them as a training data set for input into a deep learning framework; designs a first module for the initial SAR image to improve the signal-to-noise ratio before fusion, models the coherent speckle noise of the SAR image, designs a multi-scale noise suppression network based on deep learning to suppress global noise, uses a synthetic remote sensing image data set to train the network until the loss converges, and uses the Y channel of the SAR image after image quality enhancement and the visible light image as the image to be fused; designs a feature extraction module, and simultaneously designs a structure composed of alternating convolution and Transformer. Extract shallow local features and deep semantic features; the feature fusion module is divided into a Transformer fusion branch and a convolutional fusion branch, and the final fusion features are output after channel superposition; the image reconstruction module reconstructs the extracted fusion features into the final fusion image; through the above clear and concise steps, pixel-level fusion of SAR images and optical images is realized, and the feature extraction is not obvious, which avoids the problem of missing information in the fusion result, so that the information contained in the obtained fusion image is more comprehensive and accurate, suitable for various occasions, and can also be used to improve the subsequent image recognition, image tracking and other aspects caused by insufficient signal-to-noise ratio and insufficient information of SAR images in the existing technology.

[0159] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A SAR-optical image fusion method based on a hybrid model, characterized in that: include: Acquire several SAR images and optical images of the same area; Preprocessing the SAR image and the optical image to obtain a first image pair; The first image pair includes: a SAR training image and a corresponding optical training image; Acquisition of remote sensing image data; The remote sensing image data is added with noise to simulate the SAR training image to obtain a first data set; wherein the surrogate model of the noise is expressed as follows: J = ηI; J=I+(η-1)I; J = I + K; Where J is the observed image intensity, I is the noise-free image, K = (η-1)I is the multiplicative speckle component of the SAR image, η is a random sampling with mean μ and variance σ∈[0.55,0.9]; Performing end-to-end training on a multi-scale global noise suppression network according to the first data set until the training is terminated to obtain a first model; Inputting at least part of the SAR training images into the first model in batches to obtain a first input image as output; the first input image is a clean SAR image after denoising; Obtaining a second input image based on at least a portion of the grayscale channels of the optical training image; Cropping the first input image and the second input image to the same size, and forming a second image pair with images having the same content, thereby forming a second data set; An image feature extraction structure, a feature fusion structure, and a feature reconstruction structure are sequentially constructed; the image feature extraction structure, the feature fusion structure, and the feature reconstruction structure are connected to form a multi-scale global fusion model of SAR-optical images, wherein the multi-scale global fusion model is a second model, wherein the method for constructing the feature fusion structure comprises: designing a dual-branch hybrid fusion structure based on a Transformer-based fusion module and a CNN-based fusion module; wherein the Transformer fusion module mainly uses a cross-attention mechanism to complete the interaction between features, and the CNN fusion module uses the local weighted computing power of convolution and the splicing characteristics between channels to complete the interaction of features; the dual-branch hybrid fusion structure is based on a fusion method of global and local features, and finally the fused features of the two branches are spliced ​​according to channels to obtain a fusion feature map after the fusion of global and local features, thereby obtaining the feature fusion structure; According to the structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse The second model is trained until the objective function L fuse It converges to the minimum value and does not decrease any more, and the second model after training is obtained.

2. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: Preprocessing the SAR image and the optical image to obtain a first image pair includes: Optical registration and cropping are performed on the SAR image and the optical image to form the first image pair with the same size and matching content.

3. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The step of simulating the SAR image by adding noise to the remote sensing image data to obtain a first data set includes: A coherent speckle noise modeling function is obtained, and the coherent speckle noise is modeled as a noise in a product form, which is expressed as follows: y=x·n; (1) Where x is a clean grayscale image, n is the modeled noise, and y is the synthesized noise image; The SAR image is a single-view image, and n obeys a Gamma distribution with unit mean and variance 1 / L. The probability distribution of the modeling noise n is: Wherein, P(n) is the probability distribution of the modeling noise n, Γ(n) is the Gamma function, L is the number of observations, n is the modeling noise, the variance is 1 / L, and e is the base of the natural logarithm; The remote sensing image data is used as the clean grayscale image x, the clean grayscale image x is processed, the modeling noise n is added, and the noise image y is synthesized to form the first data set of the synthesized noise.

4. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The method for constructing the multi-scale global noise suppression network comprises: The encoder and decoder are designed based on the U-shaped structure. The encoder is responsible for reducing the size of the image and realizing feature extraction, and the maximum pooling layer is used to realize 2, 4, and 6 times downsampling. After each downsampling, the image size is reduced and the receptive field of the image becomes larger. The Transformer structure is used to extract global features at each size. The decoder is responsible for gradually restoring the feature map of the downsampled image with a resolution lower than the original clarity to the original input size, and then using deconvolution to realize 2, 4, and 6 times upsampling to restore the image with original clarity. After each upsampling, a Transformer structure is also used for feature recovery to restore the denoised image. At the same time, the feature maps of the same size of the encoder and the decoder are connected to obtain the multi-scale global noise suppression network.

5. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The obtaining of a second input image according to at least a portion of the grayscale channel of the optical training image comprises: The grayscale channel of the optical training image is a full-color channel of visible light. A Y channel is obtained according to the full-color channel of visible light, and the second input image is obtained according to the Y channel. The expression of the Y channel is: Y=0.257×R+0.564×G+0.098×B+16; (6) Among them, R is the red channel of the RGB color image of visible light input, G is the green channel of the RGB color image of visible light input, and B is the blue channel of the RGB color image of visible light input.

6. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The method for constructing the image feature extraction structure includes: The convolution structure and the Transformer structure are alternately mixed and connected. The convolution structure is responsible for reducing the image scale, extracting local features, and mapping the input image to a higher dimension. The Transformer structure is responsible for extracting global information, and the weight of the acquired sequence information relative to the global information is calculated according to the attention mechanism. The constructed feature extraction network is used to extract global and local features of the image to obtain the image feature extraction structure.

7. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The method for constructing the feature reconstruction structure includes: The deconvolution structure is used to restore the image that has been reduced during the feature extraction process to the same size as the input size; wherein the feature reconstruction object is the fusion feature map of the SAR image and the optical image through the feature fusion structure, which represents the comprehensive feature representation of the SAR image and optical image information. The fusion feature map is reconstructed and the abstract features are converted back to the original image space.

8. The SAR-optical image fusion method based on hybrid model according to claim 1, characterized in that: The structural similarity loss L det and feature similarity loss L feat The summation gives the objective function L fuse , according to the second data set and the objective function L fuse The second model is trained until the objective function L fuse Converging to the minimum value and no longer decreasing, the second model after training is obtained, including: The objective function L of the second model fuse Satisfies the following expression: L feat =L fuse -αL det ; (7) Among them, L feat is the feature similarity loss, L fuse is the objective function, ɑ is a constant term, 0<ɑ<1, L det is the structural similarity loss; The structural similarity loss L det The calculation formula is: L det =1-SSIM(O,I); (8) Wherein, O is the fusion result image output by the second model of the first input image and the second input image; I is the first input image input by the second model; SSIM (.) is used to measure structural similarity; The feature similarity loss L feat The expression is: Among them, w I , w1 and w2 are the trade-off parameters for balancing the loss size, both are numbers between 0 and 1, Φ f represents the feature map output by the second model; Φ1 is the optical image input by the second model, that is, the first feature map obtained by the second input image through the feature fusion structure of the second model; Φ2 is the second feature map obtained by the second input image through the feature fusion structure of the second model, and F represents the F-norm calculation.

Images