General multispectral and panchromatic image fusion method and system based on frequency domain guidance

Abstract

The invention relates to a general multispectral and panchromatic image fusion method and system based on frequency domain guidance, and the method comprises the steps: constructing a frequency domain guided multispectral and panchromatic image fusion network, comprising a spectrum constraint sub-network used for estimating fuzzy degradation of a low-resolution multispectral image and a general image fusion sub-network used for extracting spatial details of a to-be-fused image, and the general image fusion sub-network further comprises a spatial detail injection module based on frequency domain guidance. And training the frequency domain guided multispectral and panchromatic image fusion network by using the loss function of the spectrum constraint sub-network and the loss function of the general image fusion sub-network, and fusing multispectral images and panchromatic images of a plurality of satellites according to the trained multispectral and panchromatic image fusion network. Therefore, the problems that in the prior art, the degradation process does not accord with a real scene, and adaptability lacks are solved, and adaptive adjustment of the network can be achieved according to different fuzzy degrees of the input multispectral image.

Description

Technical Field

[0001] This application relates to the technical field of computer vision and deep learning, and particularly relates to a general multi-spectral and panchromatic image fusion method and system based on frequency domain guidance. Background Art

[0002] With the progress of aerospace technology, more and more satellite data has been collected, and remote sensing images have been widely used in various tasks in multiple fields such as agriculture and geological exploration, such as image segmentation and change detection. In order to ensure the effect of advanced vision tasks, the application scenarios require high-quality remote sensing satellite images. However, due to the limitations of satellite sensors, it is impossible to directly obtain high-quality multi-spectral images that simultaneously have rich spatial information and spectral information. Therefore, researchers have proposed a method of fusing low-resolution multi-spectral images with high-resolution panchromatic images to utilize the rich spatial information of the panchromatic images and the spectral characteristics of the multi-spectral images to obtain multi-spectral images with high spatial resolution.

[0003] The research on the fusion of multi-spectral images and panchromatic images has attracted a lot of attention. Common methods can be divided into methods based on component substitution (CS), methods based on multiresolution analysis (MRA), methods based on variational optimization (VO), and methods based on deep learning. Among them, both the CS-based method and the MRA-based method inject the spatial information of the panchromatic image into the low-resolution multi-spectral image to obtain a high-resolution multi-spectral image. The difference is that the CS-based method directly replaces the spatial components of the low-resolution multi-spectral image with the panchromatic image after projecting the panchromatic image and the low-resolution multi-spectral image into the transform domain to obtain a high-resolution multi-spectral image. For example, Rahmani S et al. proposed an edge-adaptive HIS (Intensity Hue Saturation) method based on the HIS method, which can reduce spectral distortion while enhancing the spatial details of the fusion result. The MRA-based method injects the spatial components of the panchromatic image into the multi-spectral image through multiresolution analysis to improve the spatial resolution of the multi-spectral image. In essence, it is also a method of information injection. Common ones include methods based on wavelet transform and methods based on Laplacian pyramid. For example, Shen et al. used the Laplacian pyramid to construct a general spatio-temporal-spectral image fusion method.

[0004] However, it is difficult to balance the spatial and spectral components in the above CS-based method and MRA-based method. To better utilize the prior information, the VO-based method physically models the problem of fusing multi-spectral images and panchromatic images to construct an energy function, introduces prior information to further compress the solution space, finally transforms the image fusion problem into an issue of optimizing the energy function, and obtains the optimal solution through iteration. For example, Liu et al. rationally designed the total variation prior term and low-rank prior term for spatial and spectral fidelity respectively, and combined the data generation fidelity term to solve and obtain high-quality fusion results. Although the above process has good theoretical support, it often requires multiple iterations, resulting in a long inference time.

[0005] In recent years, with the development and wide application of deep learning, multi-spectral image and panchromatic image fusion methods based on deep learning have been widely studied. The first deep learning-based method is PNN (Pansharpening Neural Network), which uses a convolutional neural network to extract the features of the panchromatic image and low-resolution multi-spectral image and end-to-end outputs a high-resolution multi-spectral image. To improve the effect of the deep learning multi-spectral image and panchromatic image fusion algorithm, many researchers have proposed more complex network structures for image fusion, such as MSDCNN (Multiscale and Multidepth Convolutional Neural Network), etc.

[0006] However, since high-resolution multi-spectral images cannot be obtained in real scenarios, therefore, to construct training data pairs, most of the existing deep learning-based methods are simulated through the Wald protocol. The multi-spectral image and panchromatic image are respectively subjected to blurring downsampling and downsampling operations to obtain degraded images, and the generated degraded images are used to train an end-to-end fixed neural network for multi-spectral image and panchromatic image fusion. However, most of the existing multi-spectral image and panchromatic image fusion methods are based on the degradation of the Wald protocol and all use fixed blur kernels. This degradation process is relatively simple and often does not match the degradation process in real scenarios. In addition, the existing methods do not explicitly consider the spatial degradation of the multi-spectral image to be fused and the corresponding panchromatic image feature injection weights. The fixed neural network structure often lacks adaptability and is difficult to achieve the general multi-spectral image and panchromatic image fusion task in real scenarios, which urgently needs to be solved. Summary of the Invention

[0007] The present application provides a general multi - spectral and panchromatic image fusion method and system based on frequency - domain guidance, to solve the problems existing in the prior art, such as the fixed blur kernel degradation process not conforming to the real scene, lack of self - adaptability, and difficulty in realizing general fusion tasks in real scenes. It can achieve the adaptive adjustment of the network according to the different blur degrees of the input multi - spectral image.

[0008] The first - aspect embodiment of the present application provides a general multi - spectral and panchromatic image fusion method based on frequency - domain guidance, including the following steps:

[0009] Construct a spectral constraint sub - network, where the spectral constraint sub - network is used to take a low - resolution multi - spectral image and a panchromatic image as inputs, estimate the blur degradation result of the low - resolution multi - spectral image, and construct a spectral fidelity term of the loss function according to the blur degradation result;

[0010] Construct a frequency - domain - guided spatial detail injection module, where the spatial detail injection module is used to inject the spatial details of the panchromatic image into the fusion result;

[0011] Construct a general image fusion sub - network, where the general image fusion sub - network is used to extract the spatial details of the images to be fused by using the spatial detail injection module;

[0012] Construct a frequency - domain - guided general multi - spectral and panchromatic image fusion network according to the spectral constraint sub - network and the general image fusion sub - network;

[0013] Use the loss function of the spectral constraint sub - network and the loss function of the general image fusion sub - network to train the frequency - domain - guided multi - spectral and panchromatic image fusion network, and fuse the multi - spectral images and panchromatic images of multiple satellites according to the trained multi - spectral and panchromatic image fusion network.

[0014] According to an embodiment of the present application, the spatial detail injection module includes a frequency - domain spatial detail estimation module for generating the detail injection weight of the panchromatic image, an Edge Detection module for estimating the high - frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted high - frequency details of the panchromatic image.

[0015] According to an embodiment of the present application, the general image fusion sub - network adopts a U - shaped network architecture with an encoder - decoder. The extraction of the spatial details of the images to be fused by using the spatial detail injection module includes:

[0016] Stitch the up - sampled low - resolution multi - spectral image and the panchromatic image in the channel dimension, and use the U - shaped network architecture of the encoder - decoder to extract the features of the images to be fused at multiple resolution scales;

[0017] Multiply the output feature result of the decoder by the spatial details to be injected generated by the spatial detail injection module, and connect the upsampled low-resolution multispectral image to the output end of the general image fusion sub-network in a residual connection manner to obtain the fused high-resolution multispectral image.

[0018] According to an embodiment of the present application, the encoder adopts a cascaded structure of a first preset number of ConvNext modules, and the decoder adopts a cascaded structure of a second preset number of ConvNext modules, where the first preset number is greater than the second preset number.

[0019] According to an embodiment of the present application, the loss function of the spectral constraint sub-network is:

[0020]

[0021] Y′ = ((X * C)↓ spatial * R)↓ spectral ;

[0022] Z′ = ((X * R)↓ spectral * C)↓ spatial ;

[0023] where, is the loss function of the spectral constraint sub-network, Y′ is the result after spectral degradation of the low-resolution multispectral image, Z′ is the result after spatial degradation of the panchromatic image, X is the high-resolution multispectral image, C is the blurring at the spatial level, ↓ spatial is the downsampling operation in the spatial dimension, R is the weight of each channel, ↓ spectral is the downscaling operation at the spectral level;

[0024] The loss function of the general image fusion sub-network includes a spectral constraint term and a mean absolute error loss function;

[0025] where, the spectral constraint term is:

[0026]

[0027] where, is the spectral constraint term, is the fused high-resolution multispectral image, C is the blurring at the spatial level, ↓ spatial is the downsampling operation in the spatial dimension, Y is the low-resolution multispectral image;

[0028] The mean absolute error loss function is:

[0029]

[0030] Among them, is the mean absolute error loss function, X is the high-resolution multispectral image, is the fused high-resolution multispectral image;

[0031] The loss function of the general image fusion sub-network is:

[0032]

[0033] Among them, is the loss function of the general image fusion sub-network, is the spectral constraint term, is the mean absolute error loss function.

[0034] According to the frequency-domain guided general multispectral and panchromatic image fusion method of the embodiments of the present application, a frequency-domain guided multispectral and panchromatic image fusion network is constructed. The loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network are used to train the frequency-domain guided multispectral and panchromatic image fusion network, and the multispectral images and panchromatic images of multiple satellites are fused according to the trained multispectral and panchromatic image fusion network. Thus, the problems in the prior art that the fixed blur kernel degradation process does not conform to the real scene, lacks self-adaptability, and it is difficult to achieve general fusion tasks in the real scene are solved, and the network can be adaptively adjusted according to the different blur degrees of the input multispectral images.

[0035] The second aspect of the embodiments of the present application provides a frequency-domain guided general multispectral and panchromatic image fusion system, including:

[0036] The first construction module is used to construct a spectral constraint sub-network, where the spectral constraint sub-network is used to take the low-resolution multispectral image and the panchromatic image as inputs, estimate the blur degradation result of the low-resolution multispectral image, and construct the spectral fidelity term of the loss function according to the blur degradation result;

[0037] The second construction module is used to construct a frequency-domain guided spatial detail injection module, where the spatial detail injection module is used to inject the spatial details of the panchromatic image into the fusion result;

[0038] The third construction module is used to construct a general image fusion sub-network, where the general image fusion sub-network is used to extract the spatial details of the images to be fused by using the spatial detail injection module;

[0039] The fourth construction module is used to construct a frequency-domain guided general multispectral and panchromatic image fusion network according to the spectral constraint sub-network and the general image fusion sub-network;

[0040] A fusion module, configured to train the frequency-domain guided multi-spectral and panchromatic image fusion network by using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and fuse the multi-spectral images and panchromatic images of multiple satellites according to the trained multi-spectral and panchromatic image fusion network.

[0041] According to an embodiment of the present application, the spatial detail injection module includes a frequency-domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an Edge Detection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted high-frequency details of the panchromatic image.

[0042] According to an embodiment of the present application, the general image fusion sub-network adopts a U-shaped network architecture with an encoder-decoder, and the third construction module is configured to:

[0043] Stitch the upsampled low-resolution multi-spectral image and the panchromatic image in the channel dimension, and extract features of the images to be fused at multiple resolution scales by using the U-shaped network architecture of the encoder-decoder;

[0044] Multiply the output feature result of the decoder by the spatial details to be injected generated by the spatial detail injection module, and connect the upsampled low-resolution multi-spectral image to the output end of the general image fusion sub-network in a residual connection manner to obtain a fused high-resolution multi-spectral image.

[0045] According to an embodiment of the present application, the encoder adopts a cascade structure of a first preset number of ConvNext modules, and the decoder adopts a cascade structure of a second preset number of ConvNext modules, where the first preset number is greater than the second preset number.

[0046] According to an embodiment of the present application, the loss function of the spectral constraint sub-network is:

[0047]

[0048] Y′ = ((X * C)↓ spatial * R)↓ spectral ;

[0049] Z′ = ((X * R)↓ spectral * C)↓ spatial ;

[0050] Wherein, is the loss function of the spectral constraint sub-network. Y′ is the result after spectral degradation of the low-resolution multi-spectral image, Z′ is the result after spatial degradation of the panchromatic image, X is the high-resolution multi-spectral image, C is the blurring in the spatial dimension, ↓ spatial is the downsampling operation in the spatial dimension, R is the weight of each channel, ↓ spectral is the downscaling operation in the spectral dimension;

[0051] The loss function of the general image fusion sub-network includes a spectral constraint term and a mean absolute error loss function;

[0052] Among them, the spectral constraint term is:

[0053]

[0054] Among them, is the spectral constraint term, is the fused high-resolution multi-spectral image, C is the blurring in the spatial dimension, ↓ spatial is the downsampling operation in the spatial dimension, Y is the low-resolution multi-spectral image;

[0055] The mean absolute error loss function is:

[0056]

[0057] Among them, is the mean absolute error loss function, X is the high-resolution multi-spectral image, is the fused high-resolution multi-spectral image;

[0058] The loss function of the general image fusion sub-network is:

[0059]

[0060] Among them, is the loss function of the general image fusion sub-network, is the spectral constraint term, is the mean absolute error loss function.

[0061] Based on the frequency-domain guided general multi-spectral and panchromatic image fusion system according to the embodiments of the present application, a frequency-domain guided multi-spectral and panchromatic image fusion network is constructed. The frequency-domain guided multi-spectral and panchromatic image fusion network is trained using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and the multi-spectral images and panchromatic images of multiple satellites are fused according to the trained multi-spectral and panchromatic image fusion network. Thus, the problems existing in the prior art, such as the fixed blur kernel degradation process not conforming to the real scene, lack of self-adaptability, and difficulty in realizing general fusion tasks in the real scene, are solved, and the network can be adaptively adjusted according to the different blur degrees of the input multi-spectral images.

[0062] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the frequency-domain guided general multi-spectral and panchromatic image fusion method as described in the above embodiments.

[0063] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to implement the frequency-domain guided general multi-spectral and panchromatic image fusion method as described in the above embodiments.

[0064] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present application. Description of the Drawings

[0065] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0066] Figure 1 It is a flowchart of a frequency-domain guided general multi-spectral and panchromatic image fusion method according to an embodiment of the present application;

[0067] Figure 2 It is a schematic structural diagram of the spectral constraint sub-network SimNet according to an embodiment of the present application;

[0068] Figure 3 It is a schematic structural diagram of the Edge Detection module according to an embodiment of the present application;

[0069] Figure 4 It is a schematic structural diagram of the Spatial Extraction module according to an embodiment of the present application;

[0070] Figure 5Schematic diagram of the structure of the general fusion sub-network FNet according to an embodiment of the present application;

[0071] Figure 6 Schematic diagram of the test input of the simulation and real dataset according to an embodiment of the present application;

[0072] Figure 7 Schematic diagram of the image fusion result of the simulation and real dataset according to an embodiment of the present application;

[0073] Figure 8 Schematic diagram of the comparison of the processing results of the simulation and real dataset and the results of different fusion methods according to an embodiment of the present application;

[0074] Figure 9 Block diagram of the general multi-spectral and panchromatic image fusion system based on frequency domain guidance according to an embodiment of the present application;

[0075] Figure 10 Schematic diagram of the structure of the electronic device according to an embodiment of the present application. Detailed implementation manners

[0076] The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0077] The general multi-spectral and panchromatic image fusion method and system based on frequency domain guidance according to an embodiment of the present application will be described below with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, such as the fixed blur kernel degradation process not conforming to the real scene, lack of self-adaptability, and difficulty in achieving general fusion tasks in the real scene, the present application provides a general multi-spectral and panchromatic image fusion method based on frequency domain guidance. First, in terms of data simulation, simulation data is obtained based on the Wald protocol, and anisotropic Gaussian blur with different degrees is added to the multi-spectral image to avoid the risk of model overfitting caused by single degradation. In network design, the multi-spectral and panchromatic image fusion network SBPN proposed in the present application includes a spectral constraint sub-network SimNet and a general fusion sub-network FNet.

[0078] Among them, the spectral constraint sub-network SimNet takes the low-resolution multi-spectral image and the panchromatic image as inputs, estimates the degradation of the input low-resolution multi-spectral image, and constructs a spectral fidelity constraint term for the fusion result by performing blurring downsampling on the fusion result as a loss function, making the fusion result more in line with the real situation. The general fusion sub-network FNet also includes a frequency-domain guided spatial detail injection module FGI, which can extract blurring information in the frequency domain according to the input multi-spectral image and generate a weight for injecting the panchromatic image, and can adaptively fuse the spatial details of the result. The general fusion sub-network FNet uses the ConvNext module as the feature extraction module, which can extract the feature information of the images to be fused at multiple scales. Finally, a loss function suitable for the network is designed to guide network training, and it is trained on the GaoFen-2 simulation dataset. Finally, a general multi-spectral image and panchromatic image fusion model is obtained for image denoising and super-resolution.

[0079] Specifically, Figure 1 is a schematic flow chart of a general multi-spectral and panchromatic image fusion method based on frequency-domain guidance provided by an embodiment of the present application.

[0080] As Figure 1 shown, the general multi-spectral and panchromatic image fusion method based on frequency-domain guidance includes the following steps:

[0081] In step S101, a spectral constraint sub-network is constructed. Among them, the spectral constraint sub-network is used to take the low-resolution multi-spectral image and the panchromatic image as inputs, estimate the blurring degradation result of the low-resolution multi-spectral image, and construct a spectral fidelity term of the loss function according to the blurring degradation result.

[0082] Specifically, the spectral constraint sub-network SimNet is designed using an unsupervised training method as Figure 2 shown. This network takes the low-resolution multi-spectral image and the panchromatic image as inputs, can estimate the blurring degradation of the input low-resolution multi-spectral image, and designs a spectral constraint loss function using the estimated blurring degradation to constrain the spectral consistency of the generated result.

[0083] Furthermore, assume that the input image dataset is:

[0084] D = {(Y, Z, X) i | i = 1, … N}

[0085] where N is the number of image pairs in the dataset. In this embodiment, N can be 5700, Y is the low-resolution multi-spectral image, Z is the panchromatic image, and X is the high-resolution multi-spectral image.

[0086] Therefore, the relationship between the high-resolution multi-spectral image and the low-resolution multi-spectral image can be modeled as:

[0087] Y = (X * C)↓ spatial

[0088] where C is the spatial-level blurring. In the present invention, the spatial-level blurring C is modeled as an anisotropic Gaussian blur, ↓ spatial which is a downsampling operation in the spatial dimension.

[0089] The relationship between the high-resolution multispectral image and the panchromatic image can be modeled as:

[0090] Z = (X * R)↓ spectral

[0091] where R is the weight of each channel, ↓ spectral which is a downscaling operation in the spectral dimension.

[0092] Furthermore, it is defined that Y' and Z' satisfy the following relationship:

[0093] Y' = (Y * R)↓ spectral

[0094] Z' = (Z * C)↓ spatial

[0095] where Y' is the result after spectral degradation of the low-resolution multispectral image, and Z' is the result after spatial degradation of the panchromatic image.

[0096] The above formula can be rewritten as:

[0097] T' = ((X * C)↓ spatial * R)↓ spectral

[0098] Z' = ((X * R)↓ spectral * C)↓ spatial

[0099] Therefore, according to the independence of the spatial blurring downsampling and the channel weighted downscaling operation, it can be obtained that:

[0100] Y' = Z'

[0101] That is:

[0102] (Y * R)↓ spectral = (Z * C)↓ spatial

[0103] Based on the above formula, a spectral constraint sub-network SimNet is constructed. Specifically, a convolutional layer that can be trained by the network is used as the spatial blurring C operation, an interpolation operation is used as the ↓ spatial operation, and a linear layer is used as the R and ↓ spectral operation. Therefore, this spectral constraint sub-network can be expressed as:

[0104] Y′ = Linear(Y)

[0105] Z′ = Interpolate(Conv(Z))

[0106] In step S102, a frequency-domain-guided spatial detail injection module is constructed, where the spatial detail injection module is used to inject the spatial details of the panchromatic image into the fusion result.

[0107] Specifically, a frequency-domain-guided spatial detail injection module FGI is designed. Using the luminance difference of the spectral diagrams of multi-spectral images with different blur degrees as a reference, the spatial detail injection module FGI is used as the spatial detail injection branch of the general fusion network FNet to adaptively inject the spatial details of the panchromatic image into the fusion result. Among them, the spatial detail injection module FGI takes the low-resolution multi-spectral image and the panchromatic image as inputs.

[0108] Furthermore, the frequency-domain-guided spatial detail injection module FGI includes a frequency-domain spatial detail estimation module for the low-resolution multi-spectral image, which is used to generate the detail injection weight of the panchromatic image; an Edge Detection module, which is used to estimate the high-frequency details of the panchromatic image; and a Spatial Extraction module, which is used to integrate and inject the weighted high-frequency details of the panchromatic image.

[0109] Among them, the frequency-domain spatial detail estimation module first obtains its frequency-domain information by performing a Fourier transform on the low-resolution multi-spectral image, and generates an adaptive weight map through 1 3×3 Conv and 1 Linear layer, and then obtains the final spatial detail injection weight w of the panchromatic image through average pooling:

[0110] w = AveragePool(Linear(Conv(X)))

[0111] Furthermore, as Figure 3 shown, the Edge Detection module extracts the high-frequency spatial details of the panchromatic image by using a variety of edge detection operators. In the embodiments of the present application, the Laplacian operator and the Canny operator are respectively used as the edge extraction algorithms, and the extracted high-frequency spatial details are stitched at the channel level to finally obtain the high-frequency spatial details Z of the panchromatic image h .

[0112] f laplacian = Laplacian(Z)

[0113] f canny = Canny(Z)

[0114] Zh = Concat(f laplacian , f canny )

[0115] Furthermore, as shown in Figure 4 , the Spatial Extraction module includes a convolutional layer and a Relu activation function, which can non-linearly extract the high-frequency spatial features of the panchromatic image. The obtained spatial details are injected into the weights w and the high-frequency spatial details Z of the panchromatic image h and multiplied, and the finally injected spatial features are obtained through the Spatial Extraction module. Among them, the specific form of the Spatial Extraction module is:

[0116] Z h = Conv(Conv(Relu(Conv(Z h ))))

[0117] The expression of the Relu activation function is:

[0118]

[0119] Benefiting from the design of the embodiments of the present application, in the actual application process, according to the different degrees of blurriness of the input multispectral image, the panchromatic image will be injected into the fusion result with different weights to obtain a high-quality result that can balance spectral fidelity and spatial details.

[0120] In step S103, a general image fusion sub-network is constructed, where the general image fusion sub-network is used to extract the spatial details of the images to be fused by using the spatial detail injection module.

[0121] Furthermore, in some embodiments, the general image fusion sub-network adopts a U-shaped network architecture with an encoder-decoder, and uses the spatial detail injection module to extract the spatial details of the images to be fused, including: splicing the upsampled low-resolution multispectral image and the panchromatic image in the channel dimension, and using the U-shaped network architecture of the encoder-decoder to extract the features of the images to be fused at multiple resolution scales; multiplying the output feature result of the decoder by the spatial details to be injected generated by the spatial detail injection module, and connecting the upsampled low-resolution multispectral image to the output end of the general image fusion sub-network in a residual connection manner to obtain the fused high-resolution multispectral image.

[0122] Furthermore, in some embodiments, the encoder adopts a cascaded structure of a first preset number of ConvNext modules, and the decoder adopts a cascaded structure of a second preset number of ConvNext modules, where the first preset number is greater than the second preset number.

[0123] Specifically, asFigure 5 As shown in Figure 5 , a general image fusion sub-network FNet is designed. The general image fusion sub-network FNet is mainly composed of ConvNext blocks, which are used to extract the spatial details of the images to be fused. The general image fusion network architecture adopts a U-shaped network with an encoder-decoder architecture, which is used to extract the features of the images to be fused at multiple resolution scales. At the output end of the network, the upsampled low-resolution multi-spectral image is added in the form of a residual connection to improve the learning efficiency of the network. This network takes the upsampled low-resolution multi-spectral image and the panchromatic image as inputs. In this embodiment, the upsampling factor is 4 times. At the beginning of the network, splicing is performed in the channel dimension, and features are initially extracted through a convolutional layer. This network has 3 layers. In the encoder stage, after the input features are extracted, downsampling is performed, and feature extraction is performed again at the downsampled scale. In the decoder stage, the features input to the decoder are upsampled using transposed convolution, and the output features of the encoder module at the corresponding scale are skip-connected to retain the rich feature information of the encoder.

[0124] Finally, the output result of the decoder is multiplied by the spatial details to be injected generated by the spatial detail injection module FGI after feature integration through a convolutional layer. The low-resolution multi-spectral image is connected to the output end in the form of a residual connection to further improve the spectral fidelity of the output result. Considering that the convolution kernels of typical feature extraction modules are often 3×3, the receptive field is very small, and it is difficult to achieve global attention to features. However, the Transformer architecture with global attention often has a complex spatial structure and occupies more computing resources. Therefore, this application uses the ConvNext module as the basic module for the encoder and decoder. In this embodiment, in the encoder part, 4 ConvNext modules are cascaded in each layer to fully extract features. In the decoder stage, 2 ConvNext modules are cascaded to reduce the number of network parameters while realizing the reconstruction of high-resolution multi-spectral images.

[0125] In step S104, a frequency-domain guided general multi-spectral and panchromatic image fusion network is constructed according to the spectral constraint sub-network and the general image fusion sub-network.

[0126] Specifically, a frequency-domain guided general multi-spectral and panchromatic image fusion network can be constructed according to the spectral constraint sub-network and the general image fusion sub-network. The processing process of the frequency-domain guided general multi-spectral image and panchromatic image fusion network is as follows:

[0127] First, the low-resolution multi-spectral image and the panchromatic image are input into the spectral constraint sub-network SimNet, and the blur degradation C of the input multi-spectral image is adaptively estimated through a convolutional layer and used to construct the spectral fidelity term of the loss function;

[0128] The low-resolution multispectral image and the panchromatic image are input into the general image fusion sub-network FNet. After upsampling the multispectral image and concatenating it with the panchromatic image in the channel dimension, they are input into the encoder part. The encoder extracts image features through the ConvNext module, and the features at each scale are connected to the decoder layer of the same level through residual connections; the input images pass through the spatial detail injection module FGI to obtain the spatial detail injection weights based on the frequency domain, which are multiplied with the output features of the decoder at the end of the decoder, and a residual connection of the upsampled multispectral image is added at the output end of the network. Finally, the fused high-resolution multispectral image is obtained.

[0129] In step S105, the frequency-domain guided multispectral and panchromatic image fusion network is trained using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and the multispectral images and panchromatic images of multiple satellites are fused according to the trained multispectral and panchromatic image fusion network.

[0130] Specifically, the frequency-domain guided multispectral and panchromatic image fusion network is trained in combination with the loss function, and the high-resolution multispectral image results common to multiple satellites are obtained using the trained network. Among them, the loss function adopted in the embodiments of the present application includes two parts: the loss function of the spectral constraint sub-network SimNet and the loss function of the general image fusion sub-network FNet. The loss function of the general image fusion sub-network FNet includes a spectral constraint term and the mean absolute error loss function Among them, the spectral constraint term is used to ensure the spectral consistency between the fusion result and the low-resolution multispectral image, and the mean absolute error loss function is used to ensure that the fusion result obtained by the network is close to the true value.

[0131] Among them, the loss function of the spectral constraint sub-network SimNet adopts the structural similarity loss:

[0132]

[0133] Furthermore, the spectral constraint term uses the convolutional layer weights trained by the spectral constraint sub-network SimNet to spatially downsample the fused high-resolution multispectral image and construct a loss with the input low-resolution multispectral image to constrain the spectral consistency of the fusion result.

[0134]

[0135] Among them, is the spectral constraint term, is the fused high-resolution multispectral image, C is the blurring in the spatial dimension, ↓ spatial is the downsampling operation in the spatial dimension, and Y is the low-resolution multispectral image.

[0136] Furthermore, the mean absolute error loss function is:

[0137]

[0138] where is the mean absolute error loss function, X is the high-resolution multispectral image, is the fused high-resolution multispectral image.

[0139] In the actual training process, first train the spectral constraint sub-network SimNet and obtain the convolutional layer C that can be used to estimate the blurring degradation of the low-resolution multispectral image. Subsequently, fix the parameters of the spectral constraint sub-network SimNet, and then train the general image fusion sub-network FNet.

[0140] Finally, the loss function of the general image fusion sub-network FNet is:

[0141]

[0142] where is the loss function of the general image fusion sub-network, is the spectral constraint term, is the mean absolute error loss function.

[0143] Furthermore, the embodiments of the present application can also introduce peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), spectral angle mapper (SAM), error relative global accuracy (ERGAS), and overall image quality evaluation index (Q) as evaluation indicators to measure the effect of image fusion.

[0144] The embodiments of the present application can select the GaoFen-2 satellite data as the image source of the simulation data set. Obtain the low-resolution multispectral image by adding Gaussian blurring to the multispectral image and performing downsampling in the spatial dimension. At the same time, perform downsampling on the panchromatic image to obtain the panchromatic image to be fused. Use the original multispectral image before degradation as the true value constraint training of the network.

[0145] During the network training process, this application can use the Adam optimizer, and the parameters of the optimizer are fixed as β 1 = 0.9, β 2 = 0.999. The initial learning rate is 3×10 -4 . During training, the low-resolution multispectral images in the training set are cropped to a size of 256×256×4, and the panchromatic images are cropped to a size of 1024×1024×1. 16 groups of data are selected for training in each batch, and a total of 500 rounds of training are performed.

[0146] Training is carried out on the GaoFen-2 simulation dataset, and a total of 5400 pairs of data are selected for training. The anisotropic Gaussian blur is added to the low-resolution multispectral images using the Wald protocol, with the standard deviation range of [0.2, 4] and the angle range of [0, π]. In the test stage, tests are performed on 322 pairs of simulation data from the GaoFen-2 simulation dataset and 20 pairs of real data from the QuickBird satellite respectively. The input picture examples of the two test sets are as Figure 6 shown, and the final processing results on the two datasets are as Figure 7 shown.

[0147] Based on the image fusion results obtained from the above steps, in order to compare with other methods, the Hyper-DSNet, ADKNet, MSDDN, and LDPNet in the prior art are selected as the comparison methods in the embodiments of this application to compare with the method proposed in this application, and the obtained results are as Figure 8 shown.

[0148] In order to quantitatively evaluate the results of image fusion, the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), spectral angle similarity (SAM), average global error (ERGAS), and overall image quality evaluation index (Q) are introduced as evaluation indicators in the embodiments of this application. The quantitative comparison results on the GaoFen-2 simulation dataset are shown in Table 1:

[0149] Table 1

[0150] Method Name PSNR↑ SSIM↑ ERGAS↓ Q↑ Hyper-DSNet 36.307 0.942 1.480 0.959 ADKNet 38.005 0.972 1.170 0.970 MSDDN 40.650 0.983 0.800 0.979 LDPNet 34.165 0.942 1.709 0.917 This Method 43.714 0.993 0.554 0.990

[0151] Furthermore, the quantitative comparison results on the QuickBird real dataset are shown in Table 2:

[0152] Table 2

[0153]

[0154]

[0155] According to the above quantitative index results, the image fusion results obtained by the method proposed in this application are better than the existing methods on both the simulation dataset and the real dataset, and can generate high-quality and high-resolution multispectral image fusion results on multiple satellite data.

[0156] Therefore, the present invention proposes a new frequency-domain-guided general multispectral image and panchromatic image fusion network. By introducing a frequency-guided image fusion mechanism, the network can be adaptively adjusted according to the different blurring degrees of the input multispectral images. At the same time, the ConvNext module is introduced to improve the global modeling ability of the network for the input images. The spectral constraint sub-network can adaptively estimate the blurring degradation of the multispectral image and use the estimated results to construct a spectral fidelity loss function to improve the spectral fidelity of the model output results. This method only needs to be trained on a single satellite and can be effectively generalized to multiple different satellite data. A large number of simulation and actual experimental results prove the effectiveness and practicality of the present invention.

[0157] According to the frequency-domain-guided general multispectral and panchromatic image fusion method of the embodiment of the present application, a frequency-domain-guided multispectral and panchromatic image fusion network is constructed, and the frequency-domain-guided multispectral and panchromatic image fusion network is trained using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and the multispectral images and panchromatic images of multiple satellites are fused according to the trained multispectral and panchromatic image fusion network. Thus, the problems existing in the prior art, such as the fixed blurring kernel degradation process not conforming to the real scene, lack of self-adaptability, and difficulty in realizing general fusion tasks in the real scene, are solved, and the network can be adaptively adjusted according to the different blurring degrees of the input multispectral images.

[0158] Next, a frequency-domain-guided general multispectral and panchromatic image fusion system proposed according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0159] Figure 9 It is a block diagram of the frequency-domain-guided general multispectral and panchromatic image fusion system of the embodiment of the present application.

[0160] As Figure 9 shown, the frequency-domain-guided general multispectral and panchromatic image fusion system 10 includes: a first construction module 100, a second construction module 200, a third construction module 300, a fourth construction module 400, and a fusion module 500.

[0161] Among them, the first construction module 100 is used to construct a spectral constraint sub-network, where the spectral constraint sub-network is used to take a low-resolution multi-spectral image and a panchromatic image as inputs, estimate the blurring and degradation result of the low-resolution multi-spectral image, and construct a spectral fidelity term of a loss function according to the blurring and degradation result; the second construction module 200 is used to construct a frequency-domain guided spatial detail injection module, where the spatial detail injection module is used to inject the spatial details of the panchromatic image into the fusion result; the third construction module 300 is used to construct a general image fusion sub-network, where the general image fusion sub-network is used to extract the spatial details of the images to be fused by using the spatial detail injection module; the fourth construction module 400 is used to construct a frequency-domain guided general multi-spectral and panchromatic image fusion network according to the spectral constraint sub-network and the general image fusion sub-network; the fusion module 500 is used to train the frequency-domain guided multi-spectral and panchromatic image fusion network by using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and fuse the multi-spectral images and panchromatic images of multiple satellites according to the trained multi-spectral and panchromatic image fusion network.

[0162] Further, in some embodiments, the spatial detail injection module includes a frequency-domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an Edge Detection module for estimating the high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted high-frequency details of the panchromatic image.

[0163] Further, in some embodiments, the general image fusion sub-network adopts a U-shaped network architecture with an encoder-decoder. The third construction module 300 is used to: splice the upsampled low-resolution multi-spectral image and the panchromatic image in the channel dimension, and extract the features of the images to be fused at multiple resolution scales by using the U-shaped network architecture with the encoder-decoder; multiply the output feature result of the decoder by the spatial details to be injected generated by the spatial detail injection module, and connect the upsampled low-resolution multi-spectral image to the output end of the general image fusion sub-network in a residual connection manner to obtain a fused high-resolution multi-spectral image.

[0164] Further, in some embodiments, the encoder adopts a cascaded structure of a first preset number of ConvNext modules, and the decoder adopts a cascaded structure of a second preset number of ConvNext modules, where the first preset number is greater than the second preset number.

[0165] Further, in some embodiments, the loss function of the spectral constraint sub-network is:

[0166]

[0167] Y′ = ((X * C)↓spatial *R)↓ spectral ;

[0168] Z′ = ((X * R)↓ spectral *C)↓ spatial ;

[0169] wherein, is the loss function of the spectral constraint sub-network, Y′ is the result after spectral degradation of the low-resolution multi-spectral image,, Z′ is the result after spatial degradation of the panchromatic image, X is the high-resolution multi-spectral image, C is the blurring at the spatial level, ↓ spatial is the downsampling operation in the spatial dimension, R is the weight of each channel, ↓ spectral is the downscaling operation at the spectral level;

[0170] The loss function of the general image fusion sub-network includes a spectral constraint term and a mean absolute error loss function;

[0171] wherein, the spectral constraint term is:

[0172]

[0173] wherein, is the spectral constraint term, is the fused high-resolution multi-spectral image, C is the blurring at the spatial level, ↓ spatial is the downsampling operation in the spatial dimension, Y is the low-resolution multi-spectral image;

[0174] The mean absolute error loss function is:

[0175]

[0176] wherein, is the mean absolute error loss function, X is the high-resolution multi-spectral image, is the fused high-resolution multi-spectral image;

[0177] The loss function of the general image fusion sub-network is:

[0178]

[0179] wherein, is the loss function of the general image fusion sub-network, is the spectral constraint term, is the mean absolute error loss function.

[0180] It should be noted that the foregoing explanatory description of the embodiment of the general multi-spectral and panchromatic image fusion method based on frequency-domain guidance also applies to the general multi-spectral and panchromatic image fusion system based on frequency-domain guidance in this embodiment, and will not be elaborated here.

[0181] Based on the frequency-domain guided general multi-spectral and panchromatic image fusion system according to the embodiments of the present application, a frequency-domain guided multi-spectral and panchromatic image fusion network is constructed, and the frequency-domain guided multi-spectral and panchromatic image fusion network is trained by using the loss function of the spectral constraint sub-network and the loss function of the general image fusion sub-network, and the multi-spectral images and panchromatic images of multiple satellites are fused according to the trained multi-spectral and panchromatic image fusion network. Thus, the problems existing in the prior art, such as the fixed blur kernel degradation process not conforming to the real scene, lack of self-adaptability, and difficulty in realizing the general fusion task in the real scene, are solved, and the network can be adaptively adjusted according to the different blur degrees of the input multi-spectral images.

[0182] Figure 10 The structure diagram of the electronic device provided by the embodiment of the present application. The electronic device may include:

[0183] A memory 1001, a processor 1002, and a computer program stored on the memory 1001 and executable on the processor 1002.

[0184] When the processor 1002 executes the program, it implements the frequency-domain guided general multi-spectral and panchromatic image fusion method provided in the above embodiments.

[0185] Furthermore, the electronic device further includes:

[0186] A communication interface 1003 for communication between the memory 1001 and the processor 1002.

[0187] The memory 1001 is used to store the computer program executable on the processor 1002.

[0188] The memory 1001 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0189] If the memory 1001, the processor 1002, and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001, and the processor 1002 can be interconnected through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 only a thick line is used in Figure 10 , but it does not mean that there is only one bus or one type of bus.

[0190] Optionally, in a specific implementation, if the memory 1001, the processor 1002, and the communication interface 1003 are integrated on a single chip, the memory 1001, the processor 1002, and the communication interface 1003 can communicate with each other through an internal interface.

[0191] The processor 1002 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0192] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned method for fusing general multi-spectral and panchromatic images based on frequency-domain guidance is implemented.

[0193] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or N embodiments or examples. In addition, without conflict, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0194] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0195] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A general multispectral and panchromatic image fusion method based on frequency domain guidance, characterized in that: The following steps are involved: Constructing a spectral constraint subnetwork, wherein the spectral constraint subnetwork is used to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result; Constructing a frequency domain guided spatial detail injection module, wherein the spatial detail injection module is used to inject the spatial details of the full color image into the fusion result; Constructing a general image fusion subnetwork, wherein the general image fusion subnetwork is used to extract spatial details of the image to be fused using the spatial detail injection module; Constructing a frequency domain guided universal multi-spectral and panchromatic image fusion network according to the spectral constraint subnetwork and the universal image fusion subnetwork; The frequency-domain guided multispectral and panchromatic image fusion network is trained using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and the multispectral images and panchromatic images of multiple satellites are fused based on the trained multispectral and panchromatic image fusion network.

2. The method according to claim 1, characterized in that The spatial detail injection module includes a frequency domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an EdgeDetection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting weighted high-frequency details of the panchromatic image.

3. The method according to claim 2, characterized in that The general image fusion sub-network adopts a U-shaped network architecture with an encoder-decoder, and the spatial detail injection module is used to extract the spatial details of the image to be fused, including: The upsampled low-resolution multispectral image and the panchromatic image are spliced ​​in the channel dimension, and the features of the image to be fused are extracted at multiple resolution scales using the U-shaped network architecture of the encoder-decoder; The output feature result of the decoder is multiplied by the spatial detail to be injected generated by the spatial detail injection module, and the upsampled low-resolution multispectral image is connected to the output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

4. The method according to claim 3, characterized in that The encoder adopts a cascade structure of ConvNext modules with a first preset number, and the decoder adopts a cascade structure of ConvNext modules with a second preset number, wherein the first preset number is greater than the second preset number.

5. The method according to claim 1, characterized in that The loss function of the spectral constraint subnetwork is: Y′=((X*C)↓ spatial *R)↓ spectral ; Z′=((X*R)↓ spectral *C)↓ spatial ; in, is the loss function of the spectral constraint subnetwork, Y′ is the result of spectral degradation of the low-resolution multispectral image, Z′ is the result of spatial degradation of the panchromatic image, X is the high-resolution multispectral image, C is the blur at the spatial level, ↓ spatial is the downsampling operation of the spatial dimension, R is the weight of each channel, ↓ spectral It is a downscaling operation at the spectral level; The loss function of the general image fusion subnetwork includes a spectral constraint term and a mean absolute error loss function; Wherein, the spectral constraint term is: in, is the spectral constraint term, is the high-resolution multispectral image obtained by fusion, C is the blur at the spatial level, ↓ spatial is the downsampling operation of the spatial dimension, and Y is the low-resolution multispectral image; The mean absolute error loss function is: in, is the mean absolute error loss function, X is a high-resolution multispectral image, The high-resolution multispectral image is obtained by fusion; The loss function of the general image fusion subnetwork is: in, is the loss function of the general image fusion subnetwork, is the spectral constraint term, is the mean absolute error loss function.

6. A general multispectral and panchromatic image fusion system based on frequency domain guidance, characterized in that: include: A first construction module is used to construct a spectral constraint sub-network, wherein the spectral constraint sub-network is used to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result; A second construction module is used to construct a frequency domain guided spatial detail injection module, wherein the spatial detail injection module is used to inject the spatial details of the full color image into the fusion result; A third construction module is used to construct a general image fusion sub-network, wherein the general image fusion sub-network is used to extract the spatial details of the image to be fused by using the spatial detail injection module; A fourth construction module is used to construct a frequency domain guided universal multi-spectral and panchromatic image fusion network according to the spectral constraint sub-network and the universal image fusion sub-network; A fusion module is used to train the frequency-domain guided multispectral and panchromatic image fusion network using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and to fuse the multispectral images and panchromatic images of multiple satellites according to the trained multispectral and panchromatic image fusion network.

7. The system according to claim 6, characterized in that The spatial detail injection module includes a frequency domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an EdgeDetection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting weighted high-frequency details of the panchromatic image.

8. The system according to claim 7, characterized in that The general image fusion sub-network adopts a U-shaped network architecture with an encoder-decoder, and the third building block is used to: The upsampled low-resolution multispectral image and the panchromatic image are spliced ​​in the channel dimension, and the features of the image to be fused are extracted at multiple resolution scales using the U-shaped network architecture of the encoder-decoder; The output feature result of the decoder is multiplied by the spatial detail to be injected generated by the spatial detail injection module, and the upsampled low-resolution multispectral image is connected to the output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the general multispectral and panchromatic image fusion method based on frequency domain guidance as described in any one of claims 1 to 5.

10. A computer storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the general multispectral and panchromatic image fusion method based on frequency domain guidance as described in any one of claims 1 to 5.

Images