General multispectral and panchromatic image fusion method and system based on frequency domain guidance

Abstract

The application relates to a general multispectral and panchromatic image fusion method and system based on frequency domain guidance, which comprises the following steps: constructing a frequency domain guided multispectral and panchromatic image fusion network, including a spectral constraint subnetwork for estimating low-resolution multispectral image blur degradation and a general image fusion subnetwork for extracting spatial details of to-be-fused images, the general image fusion subnetwork further comprising a spatial detail injection module based on frequency domain guidance, training the frequency domain guided multispectral and panchromatic image fusion network by using a loss function of the spectral constraint subnetwork and a loss function of the general image fusion subnetwork, and fusing multispectral images and panchromatic images of multiple satellites according to the trained multispectral and panchromatic image fusion network. Therefore, the problems of inconsistency between the degradation process and the real scene and lack of adaptability in the prior art are solved, and adaptive adjustment of the network can be realized according to different blur degrees of input multispectral images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a general multispectral and panchromatic image fusion method and system based on frequency domain guidance. BACKGROUND

[0002] With the progress of aviation technology, more and more satellite data is collected, and remote sensing images are widely used in various tasks in the fields of agriculture, geological exploration and the like, such as image segmentation, change detection and the like. In order to ensure the effect of high-level vision tasks, high-quality remote sensing satellite images are required in application scenarios. However, due to the limitation of satellite sensors, high-quality multispectral images with rich spatial information and spectral information cannot be directly obtained. Therefore, researchers have proposed a method of fusing low-resolution multispectral images and high-resolution panchromatic images to utilize the rich spatial information of panchromatic images and the spectral characteristics of multispectral images, so as to obtain high spatial resolution multispectral images.

[0003] Research on the fusion of multispectral images and panchromatic images has attracted a lot of attention, and common methods can be divided into component substitution (CS) based methods, multiresolution analysis (MRA) based methods, variational optimization (VO) based methods and deep learning based methods. Among them, the CS based method and the MRA based method are both methods of injecting spatial information of panchromatic images into low-resolution multispectral images to obtain high-resolution multispectral images. The difference is that the CS based method directly uses the panchromatic image to replace the spatial component of the low-resolution multispectral image to obtain the high-resolution multispectral image by projecting the panchromatic image and the low-resolution multispectral image into the transform domain. For example, Rahmani S et al. proposed an edge adaptive IHS method based on the HIS (Intensity Hue Saturation) method, which can improve the spatial details of the fusion result while reducing spectral distortion. The MRA based method injects the spatial component of the panchromatic image into the multispectral image to improve the spatial resolution of the multispectral image by the method of multiresolution analysis, which is essentially also an information injection method. Common methods include wavelet transform based methods and Laplacian pyramid based methods, etc. For example, Shen et al. constructed a general spatial-temporal-spectral image fusion method based on the Laplacian pyramid.

[0004] However, the above CS-based method and MRA-based method are difficult to balance spatial components and spectral components. In order to better utilize prior information, the VO-based method constructs an energy function by physically modeling the multispectral image and panchromatic image fusion problem, and introduces prior information to further compress the solution space, finally converts the image fusion problem into an optimization problem of the energy function, and obtains the optimal solution through iteration. For example, Liu et al. designed total variation prior terms and low rank prior terms for spatial and spectral preservation respectively, and combined data generation fidelity terms to obtain a high-quality fusion result. Although the above process has good theoretical support, it often needs multiple iterations, resulting in long reasoning time.

[0005] In recent years, with the development and wide application of deep learning, deep learning-based multispectral image and panchromatic image fusion methods have been widely studied. The first deep learning-based method is PNN (Pansharpening Neural Network), which uses a convolutional neural network to extract features of the panchromatic image and the low-resolution multispectral image and outputs a high-resolution multispectral image end-to-end. In order to improve the effect of deep learning-based multispectral image and panchromatic image fusion algorithm, many researchers have proposed more complex network structures for image fusion, such as MSDCNN (Multiscale and Multidepth Convolutional Neural Network).

[0006] However, since high-resolution multispectral images cannot be obtained in real scenes, in order to construct a training data pair, most existing deep learning-based methods use the Wald protocol for simulation, and perform blurring down-sampling and down-sampling on the multispectral image and the panchromatic image respectively to obtain degraded images, and use the generated degraded images to train an end-to-end fixed neural network for multispectral image and panchromatic image fusion. However, most existing multispectral image and panchromatic image fusion methods are based on Wald protocol degradation, and use fixed blur kernels. This degradation process is relatively simple and often does not match the degradation process in real scenes. In addition, existing methods do not explicitly consider the spatial degradation of the multispectral image to be fused and the matching panchromatic image feature injection weight. The fixed neural network structure often lacks adaptability and is difficult to achieve a general multispectral image and panchromatic image fusion task in real scenes, which needs to be solved urgently. SUMMARY

[0007] The application provides a general multispectral and panchromatic image fusion method and system based on frequency domain guidance to solve the problems of fixed blur kernel degradation process, lack of adaptability and difficulty in realizing general fusion tasks in real scenes in the prior art, and can realize adaptive adjustment of the network according to different blur degrees of input multispectral images.

[0008] The first aspect embodiment of the application provides a general multispectral and panchromatic image fusion method based on frequency domain guidance, comprising the following steps:

[0009] A spectral constraint subnetwork is constructed, wherein the spectral constraint subnetwork is used to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result;

[0010] A spatial detail injection module based on frequency domain guidance is constructed, wherein the spatial detail injection module is used to inject spatial details of the panchromatic image into a fusion result;

[0011] A general image fusion subnetwork is constructed, wherein the general image fusion subnetwork is used to extract spatial details of to-be-fused images by using the spatial detail injection module;

[0012] A frequency domain guided general multispectral and panchromatic image fusion network is constructed according to the spectral constraint subnetwork and the general image fusion subnetwork;

[0013] The frequency domain guided multispectral and panchromatic image fusion network is trained by using a loss function of the spectral constraint subnetwork and a loss function of the general image fusion subnetwork, and multispectral images and panchromatic images of multiple satellites are fused according to the trained multispectral and panchromatic image fusion network.

[0014] According to one embodiment of the application, the spatial detail injection module includes a frequency domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an Edge Detection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted panchromatic image high-frequency details.

[0015] According to one embodiment of the application, the general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and the extraction of spatial details of to-be-fused images by using the spatial detail injection module comprises:

[0016] The up-sampled low-resolution multispectral image and the panchromatic image are spliced in the channel dimension, and the U-shaped network architecture of the encoder-decoder is used to extract features of the to-be-fused images at multiple resolution scales.

[0017] The output feature result of the decoder is multiplied by the to-be-injected spatial details generated by the spatial detail injection module, and the up-sampled low-resolution multispectral image is connected to the output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

[0018] According to an embodiment of the present application, the encoder adopts a first preset number of ConvNext module cascaded structures, and the decoder adopts a second preset number of ConvNext module cascaded structures, wherein the first preset number is greater than the second preset number.

[0019] According to an embodiment of the present application, the loss function of the spectral constraint subnetwork is:

[0020]

[0021] Y' = ((X * C) down spatial *R) down spectral ;

[0022] Z' = ((X * R) down spectral *C) down spatial ;

[0023] wherein, is the loss function of the spectral constraint subnetwork, Y' is the result of spectral degradation of a low-resolution multispectral image, Z' is the result of spatial degradation of a panchromatic image, X is a high-resolution multispectral image, C is a blur in the spatial dimension, down spatial is a down-sampling operation in the spatial dimension, R is a weight of each channel, down spectral is a down-scaling operation in the spectral dimension;

[0024] The loss function of the general image fusion subnetwork includes a spectral constraint term and a mean absolute error loss function;

[0025] wherein, the spectral constraint term is:

[0026]

[0027] wherein, is the spectral constraint term, is a fused high-resolution multispectral image, C is a blur in the spatial dimension, down spatial is a down-sampling operation in the spatial dimension, Y is a low-resolution multispectral image;

[0028] The mean absolute error loss function is:

[0029]

[0030] wherein, is a mean absolute error loss function, X is a high-resolution multispectral image, is a fused high-resolution multispectral image;

[0031] The loss function of the general image fusion subnetwork is:

[0032]

[0033] wherein, is a loss function of the general image fusion subnetwork, is a spectral constraint term, is a mean absolute error loss function.

[0034] According to the general multispectral and panchromatic image fusion method based on frequency domain guidance provided in the embodiments of the present application, a multispectral and panchromatic image fusion network based on frequency domain guidance is constructed, the multispectral and panchromatic image fusion network based on frequency domain guidance is trained by using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and the multispectral and panchromatic image fusion network based on frequency domain guidance is used to fuse the multispectral images and panchromatic images of multiple satellites. In this way, the problems existing in the prior art, such as the fixed blur kernel degradation process not consistent with the real scene, lack of adaptability, and difficulty in realizing the general fusion task in the real scene, are solved, and the adaptive adjustment of the network can be realized according to different blur degrees of the input multispectral images.

[0035] The second aspect of the embodiments of the present application provides a general multispectral and panchromatic image fusion system based on frequency domain guidance, comprising:

[0036] A first construction module is configured to construct a spectral constraint subnetwork, wherein the spectral constraint subnetwork is configured to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result;

[0037] A second construction module is configured to construct a spatial detail injection module based on frequency domain guidance, wherein the spatial detail injection module is configured to inject spatial details of the panchromatic image into a fusion result;

[0038] A third construction module is configured to construct a general image fusion subnetwork, wherein the general image fusion subnetwork is configured to extract spatial details of to-be-fused images by using the spatial detail injection module;

[0039] A fourth construction module is configured to construct a general multispectral and panchromatic image fusion network based on frequency domain guidance according to the spectral constraint subnetwork and the general image fusion subnetwork;

[0040] The fusion module is used to train the frequency-domain guided multispectral and panchromatic image fusion network using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and to fuse multispectral and panchromatic images from multiple satellites based on the trained multispectral and panchromatic image fusion network.

[0041] According to one embodiment of this application, the spatial detail injection module includes a frequency domain spatial detail estimation module for generating detail injection weights for the panchromatic image, an Edge Detection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting high-frequency details of the weighted panchromatic image.

[0042] According to one embodiment of this application, the general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and the third construction module is used for:

[0043] The upsampled low-resolution multispectral image and the panchromatic image are stitched together along the channel dimension, and the features of the image to be fused are extracted at multiple resolution scales using the encoder-decoder U-shaped network architecture.

[0044] The output feature result of the decoder is multiplied with the spatial details to be injected generated by the spatial detail injection module, and the upsampled low-resolution multispectral image is connected to the output of the general image fusion subnetwork in the form of residual connection to obtain the fused high-resolution multispectral image.

[0045] According to one embodiment of this application, the encoder adopts a first preset number of ConvNext module cascaded structure, and the decoder adopts a second preset number of ConvNext module cascaded structure, wherein the first preset number is greater than the second preset number.

[0046] According to one embodiment of this application, the loss function of the spectral constraint subnetwork is:

[0047]

[0048] Y′=((X*C)↓ spatial *R)↓ spectral ;

[0049] Z′=((X*R)↓ spectral *C)↓ spatial ;

[0050] in, For the loss function of the spectral constraint subnetwork, Y' is the result of spectral degradation of the low-resolution multispectral image, Z' is the result of spatial degradation of the panchromatic image, X is the high-resolution multispectral image, C is the blur in the spatial dimension, and spatial is a down-sampling operation in the spatial dimension, R is the weight of each channel, and spectral is a down-scaling operation in the spectral dimension;

[0051] The loss function of the general image fusion subnetwork includes a spectral constraint term and a mean absolute error loss function;

[0052] The spectral constraint term is:

[0053]

[0054] wherein, is the spectral constraint term, is the fused high-resolution multispectral image, C is the blur in the spatial dimension, and spatial is a down-sampling operation in the spatial dimension, Y is the low-resolution multispectral image;

[0055] The mean absolute error loss function is:

[0056]

[0057] wherein, is the mean absolute error loss function, X is the high-resolution multispectral image, is the fused high-resolution multispectral image;

[0058] The loss function of the general image fusion subnetwork is:

[0059]

[0060] wherein, is the loss function of the general image fusion subnetwork, is the spectral constraint term, is the mean absolute error loss function.

[0061] According to the universal multispectral and panchromatic image fusion system based on frequency domain guidance provided by the embodiment of the present application, a frequency domain guidance multispectral and panchromatic image fusion network is constructed, the frequency domain guidance multispectral and panchromatic image fusion network is trained by using a loss function of a spectral constraint subnetwork and a loss function of a universal image fusion subnetwork, and the multispectral and panchromatic image fusion network after training is used to fuse multispectral images and panchromatic images of multiple satellites. Thus, the problems in the prior art, such as the fixed blur kernel degradation process not consistent with the real scene, lack of adaptability, and difficulty in realizing the universal fusion task in the real scene, are solved, and the adaptive adjustment of the network can be realized according to different blur degrees of the input multispectral images.

[0062] The third aspect embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the universal multispectral and panchromatic image fusion method based on frequency domain guidance as described in the above embodiments.

[0063] The fourth aspect embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the universal multispectral and panchromatic image fusion method based on frequency domain guidance as described in the above embodiments.

[0064] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0065] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings.

[0066] Figure 1 A flowchart of a universal multispectral and panchromatic image fusion method based on frequency domain guidance according to an embodiment of the present application;

[0067] Figure 2 A structural diagram of a spectral constraint subnetwork SimNet according to an embodiment of the present application;

[0068] Figure 3 A structural diagram of an Edge Detection module according to an embodiment of the present application;

[0069] Figure 4 A structural diagram of a Spatial Extraction module according to an embodiment of the present application;

[0070] Figure 5Fig. 1 is a structural schematic diagram of a general-purpose fusion subnetwork FNet according to an embodiment of the present application;

[0071] Figure 6 Fig. 2 is a schematic diagram of simulation and real dataset test input according to an embodiment of the present application;

[0072] Figure 7 Fig. 3 is a schematic diagram of image fusion results of simulation and real dataset according to an embodiment of the present application;

[0073] Figure 8 Fig. 4 is a schematic diagram of processing results of simulation and real dataset and results of different fusion methods according to an embodiment of the present application;

[0074] Figure 9 Fig. 5 is a block schematic diagram of a general-purpose multispectral and panchromatic image fusion system based on frequency domain guidance according to an embodiment of the present application;

[0075] Figure 10 Fig. 6 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] Embodiments of the present application are described in detail below with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0077] A general-purpose multispectral and panchromatic image fusion method and system based on frequency domain guidance according to an embodiment of the present application is described below with reference to the accompanying drawings. In view of the problems mentioned in the background art, such as the fixed blur kernel degradation process not consistent with real scenes, lack of adaptability, and difficulty in implementing general-purpose fusion tasks in real scenes, the present application provides a general-purpose multispectral and panchromatic image fusion method based on frequency domain guidance. First, in terms of data simulation, simulation data is obtained based on the Wald protocol, and different degrees of anisotropic Gaussian blur are added to the multispectral image to avoid the risk of model overfitting caused by single degradation. In network design, the multispectral and panchromatic image fusion network SBPN proposed in the present application includes a spectral constraint subnetwork SimNet and a general-purpose fusion subnetwork FNet.

[0078] The spectral constraint subnetwork SimNet takes a low-resolution multispectral image and a panchromatic image as input, estimates the degradation of the input low-resolution multispectral image, and performs fuzzy down-sampling on the fusion result as a loss function to construct a spectral fidelity constraint term, so that the fusion result is more consistent with the actual situation. The spatial detail injection module FGI based on frequency domain guidance is also included in the general fusion subnetwork FNet, which can extract fuzzy information in the frequency domain according to the input multispectral image and generate panchromatic image injection weights, and can adaptively fuse the spatial details of the fusion result. The general fusion subnetwork FNet uses the ConvNext module as a feature extraction module, which can extract feature information of the images to be fused at multiple scales. Finally, a loss function suitable for the network is designed to guide the network training, and the general multispectral image and panchromatic image fusion model is obtained by training on the GaoFen-2 simulation dataset, and image denoising and super-resolution are performed.

[0079] Specifically, Figure 1 A flowchart of a general multispectral and panchromatic image fusion method based on frequency domain guidance is provided for the embodiments of the present application.

[0080] As Figure 1 shown, the general multispectral and panchromatic image fusion method based on frequency domain guidance includes the following steps:

[0081] In step S101, a spectral constraint subnetwork is constructed, wherein the spectral constraint subnetwork is used to take a low-resolution multispectral image and a panchromatic image as input, estimate the fuzzy degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of the loss function according to the fuzzy degradation result.

[0082] Specifically, the spectral constraint subnetwork SimNet is designed using an unsupervised training method as shown in Figure 2 The network takes a low-resolution multispectral image and a panchromatic image as input, can estimate the fuzzy degradation of the input low-resolution multispectral image, and uses the estimated fuzzy degradation to design a spectral constraint loss function to constrain the spectral consistency of the generated result.

[0083] Further, it is assumed that the input image dataset is:

[0084] D={(Y,Z,X) i |i=1,…N}

[0085] Where N is the number of image pairs in the dataset, in this embodiment, N can be 5700, Y is a low-resolution multispectral image, Z is a panchromatic image, and X is a high-resolution multispectral image.

[0086] Therefore, the relationship between the high-resolution multispectral image and the low-resolution multispectral image can be modeled as:

[0087] Y = (X * C) ↓ spatial

[0088] where C is the spatial level blur, in the present disclosure, the spatial level blur C is modeled as an anisotropic Gaussian blur, ↓ spatial is a down-sampling operation in spatial dimension.

[0089] The relationship between the high-resolution multispectral image and the panchromatic image can be modeled as:

[0090] Z = (X * R) ↓ spectral

[0091] where R is the weight of each channel, ↓ spectral is a down-scaling operation in spectral dimension.

[0092] Further, it is defined that Y' and Z' satisfy the following relationship:

[0093] Y' = (Y * R) ↓ spectral

[0094] Z' = (Z * C) ↓ spatial

[0095] where Y' is the result of the low-resolution multispectral image after spectral degradation, and Z' is the result of the panchromatic image after spatial degradation.

[0096] The above formula can be rewritten as:

[0097] T' = ((X * C) ↓ spatial *R) ↓ spectral

[0098] Z' = ((X * R) ↓ spectral *C) ↓ spatial

[0099] Therefore, according to the independence of the spatial blur down-sampling and the channel weighted down-scaling operation, it can be obtained that:

[0100] Y' = Z'

[0101] That is:

[0102] (Y * R) ↓ spectral = (Z * C) ↓ spatial

[0103] According to the above formula, a spectral constraint sub-network SimNet is constructed. Specifically, a convolution layer that can be trained by the network is used as the spatial blur C operation, an interpolation operation is used as the ↓ spatial operation, and a linear layer is used as the R and ↓ spectral operation. Therefore, the spectral constraint sub-network can be expressed as:

[0104] Y' = Linear(Y)

[0105] Z' = Interpolate(Conv(Z))

[0106] In step S102, a spatial detail injection module based on frequency domain guidance is constructed, wherein the spatial detail injection module is used to inject spatial details of the panchromatic image into the fusion result.

[0107] Specifically, a spatial detail injection module FGI based on frequency domain guidance is designed, which uses the brightness difference of the spectral graph of the multispectral image with different blur degrees as a reference, and injects spatial details of the panchromatic image into the fusion result adaptively by taking the spatial detail injection module FGI as a spatial detail injection branch of the general fusion network FNet. Wherein, the spatial detail injection module FGI takes the low-resolution multispectral image and the panchromatic image as input.

[0108] Further, the spatial detail injection module FGI based on frequency domain guidance includes a frequency domain spatial detail estimation module for the low-resolution multispectral image, which is used to generate detail injection weights of the panchromatic image; an Edge Detection module for estimating high-frequency details of the panchromatic image; a Spatial Extraction module for integrating and injecting the weighted high-frequency details of the panchromatic image.

[0109] Wherein, the frequency domain spatial detail estimation module obtains the frequency domain information of the low-resolution multispectral image by first passing it through the Fourier transform, and generates adaptive weight maps through a 3x3 Conv and a Linear layer, and then obtains the final panchromatic image spatial detail injection weight w through average pooling:

[0110] w = AveragePool(Linear(Conv(X)))

[0111] Further, as shown in Figure 3 , the Edge Detection module extracts high-frequency spatial details of the panchromatic image by using a variety of edge detection operators, and the Laplacian operator and the Canny operator are used as edge extraction algorithms, and the extracted high-frequency spatial details are spliced at the channel level to obtain the high-frequency spatial details Z h of the panchromatic image.

[0112] f laplacian = Laplacian(Z)

[0113] f canny = Canny(Z)

[0114] Zh =Concat(f laplacian ,f canny )

[0115] Furthermore, such as Figure 4 As shown, the Spatial Extraction module contains convolutional layers and a ReLU activation function, enabling non-linear extraction of high-frequency spatial features from panchromatic images. The obtained spatial details are then injected into weights w and the high-frequency spatial details Z of the panchromatic image. h The features are multiplied and then processed through the Spatial Extraction module to obtain the final injected spatial features. The Spatial Extraction module takes the following form:

[0116] Z h =Conv(Conv(Relu(Conv(Z)) h ))))

[0117] The ReLU activation function expression is:

[0118]

[0119] Thanks to the design of the embodiments of this application, in practical applications, depending on the degree of blur of the input multispectral image, the panchromatic image will be injected into the fusion result with different weights to obtain a high-quality result that can balance spectral fidelity and spatial detail.

[0120] In step S103, a general image fusion subnetwork is constructed, wherein the general image fusion subnetwork is used to extract spatial details of the image to be fused using the spatial detail injection module.

[0121] Furthermore, in some embodiments, the general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and uses a spatial detail injection module to extract spatial details of the image to be fused, including: stitching the upsampled low-resolution multispectral image and the panchromatic image in the channel dimension; using the encoder-decoder U-shaped network architecture to extract features of the image to be fused at multiple resolution scales; multiplying the output feature result of the decoder with the spatial details to be injected generated by the spatial detail injection module, and connecting the upsampled low-resolution multispectral image to the output of the general image fusion subnetwork in a residual connection manner to obtain the fused high-resolution multispectral image.

[0122] Furthermore, in some embodiments, the encoder adopts a first preset number of ConvNext module cascaded structure, and the decoder adopts a second preset number of ConvNext module cascaded structure, wherein the first preset number is greater than the second preset number.

[0123] Specifically, such asFigure 5 As shown, a general image fusion subnetwork FNet is designed. The general image fusion subnetwork FNet is mainly composed of ConvNext blocks, which are used to extract spatial details of the images to be fused. The general image fusion network architecture adopts a U-shaped network with an encoder-decoder architecture, which is used to extract features of the images to be fused at multiple resolution scales. A low-resolution multispectral image after upsampling is added in a residual connection manner at the output end of the network to improve the learning efficiency of the network. The network takes the low-resolution multispectral image after upsampling and the panchromatic image as input. In this embodiment, the upsampling multiple is 4. The network starts to splice in the channel dimension, and preliminarily extracts features through the convolution layer. The network has 3 layers in total. In the encoder stage, the input features are downsampled after feature extraction, and feature extraction is performed again at the scale after downsampling. In the decoder stage, the features input into the decoder are upsampled using transposed convolution, and the output features of the encoder module at the corresponding scale are connected in a jump connection manner to retain the rich feature information of the encoder.

[0124] Finally, the output result of the decoder is multiplied by the spatial details to be injected generated by the spatial detail injection module FGI after feature integration by the convolution layer, and the low-resolution multispectral image is connected to the output end in a residual connection manner to further improve the spectral fidelity of the output result. Considering that the convolution kernel of a typical feature extraction module is usually 3x3, the receptive field is small, and it is difficult to extract global attention to the features, and the Transformer architecture with global attention usually has a complex spatial structure and occupies more computing resources, therefore, the ConvNext module is adopted as the basic module of the encoder and the decoder. In this embodiment, the encoder part adopts a cascaded manner of 4 ConvNext modules per layer to fully extract features, and the decoder part adopts a cascaded manner of 2 ConvNext modules to realize high-resolution multispectral image reconstruction while reducing the network parameter amount.

[0125] In step S104, a frequency domain guided general multispectral and panchromatic image fusion network is constructed according to the spectral constraint subnetwork and the general image fusion subnetwork.

[0126] Specifically, the frequency domain guided general multispectral and panchromatic image fusion network can be constructed according to the spectral constraint subnetwork and the general image fusion subnetwork, wherein the processing process of the frequency domain guided general multispectral and panchromatic image fusion network is as follows:

[0127] Firstly, the low-resolution multispectral image and the panchromatic image are input into the spectral constraint subnetwork SimNet, and the blur degradation C of the input multispectral image is adaptively estimated through the convolution layer and used to construct the spectral fidelity term of the loss function;

[0128] The low-resolution multispectral image and the panchromatic image are input into a general image fusion subnetwork FNet, the multispectral image is upsampled and spliced with the panchromatic image in the channel dimension, and then input into an encoder part, the encoder extracts image features through a ConvNext module, and the features at different scales are connected through a residual connection and input into the same level decoder layer; the input image is input into a spatial detail injection module FG I to obtain a spatial detail injection weight based on a frequency domain, the spatial detail injection weight is multiplied with the output feature of the decoder at the end of the decoder, and a residual connection of the upsampled multispectral image is added at the output end of the network, and finally, a fused high-resolution multispectral image is obtained.

[0129] In step S105, the frequency domain guided multispectral and panchromatic image fusion network is trained by using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and the multispectral images and the panchromatic images of a plurality of satellites are fused according to the multispectral and panchromatic image fusion network after the training is completed.

[0130] Specifically, the frequency domain guided multispectral and panchromatic image fusion network is trained in combination with the loss function, and the trained network is used to obtain a high-resolution multispectral image result of a plurality of satellites. The loss function used in the embodiment of the application includes two parts: a loss function of a spectral constraint subnetwork SimNet and a loss function of a general image fusion subnetwork FNet. The loss function of the general image fusion subnetwork FNet includes a spectral constraint term and a mean absolute error loss function The spectral constraint term is used to ensure the spectral consistency of the fusion result and the low-resolution multispectral image, and the mean absolute error loss function is used to ensure that the fusion result obtained by the network is close to the true value.

[0131] The loss function of the spectral constraint subnetwork SimNet uses a structural similarity loss:

[0132]

[0133] Further, the spectral constraint term uses the convolution layer weight obtained by training the spectral constraint subnetwork SimNet to perform spatial down-sampling on the fused high-resolution multispectral image, and constructs a loss with the input low-resolution multispectral image, which is used to constrain the spectral consistency of the fusion result.

[0134]

[0135] wherein, is the spectral constraint term, The high-resolution multispectral image obtained by fusion is C, which is a blur in the spatial dimension, spatial Y is a low-resolution multispectral image.

[0136] Further, the mean absolute error loss function is:

[0137]

[0138] wherein, is the mean absolute error loss function, X is a high-resolution multispectral image, is a high-resolution multispectral image obtained by fusion.

[0139] In the actual training process, first, the spectral constraint subnetwork SimNet is trained, and a convolutional layer C that can be used to estimate the blur degradation of a low-resolution multispectral image is obtained, and then the parameters of the spectral constraint subnetwork SimNet are fixed, and the general image fusion subnetwork FNet is trained.

[0140] Finally, the loss function of the general image fusion subnetwork FNet is:

[0141]

[0142] wherein, is the loss function of the general image fusion subnetwork, is a spectral constraint term, is a mean absolute error loss function.

[0143] Further, the peak signal-to-noise ratio (PSNR), the structural similarity index measure (SSIM), the spectral angle mapper (SAM), the error relative global accuracy (ERGAS), and the image overall quality evaluation index (Q) can be introduced as evaluation indexes by the embodiments of the present application, which are used to measure the effect of image fusion.

[0144] The embodiments of the present application can select GaoFen-2 satellite data as the image source of the simulation data set, obtain a low-resolution multispectral image by adding Gaussian blur to the multispectral image and performing down-sampling in the spatial dimension, and obtain a panchromatic image to be fused by performing down-sampling on the panchromatic image. The original multispectral image before degradation is used as the true value constraint for training the network.

[0145] In the network training process, the present application can use the Adam optimizer, and the parameters of the optimizer are fixed as β1=0.9, β2=0.999. The initial learning rate is 3×10 -4 During training, the low-resolution multispectral images in the training set are cropped to 256×256×4 size, and the panchromatic image is cropped to 1024×1024×1 size. 16 groups of data are selected for training in each batch, and a total of 500 rounds of training are performed.

[0146] Training is performed on the GaoFen-2 simulation dataset, and a total of 5400 pairs of data are selected for training. The low-resolution multispectral images are added with anisotropic Gaussian blur using the Wald protocol, with a standard deviation range of [0.2, 4] and an angle range of [0, π]. In the test phase, 322 pairs of simulation data from the GaoFen-2 simulation dataset and 20 pairs of real data from the QuickBird satellite are tested, and the input image samples of the two test sets are as shown in Figure 6 , and the final processing results on the two datasets are as shown in Figure 7 .

[0147] Based on the image fusion results obtained by the above steps, in order to compare with other methods, the present application embodiment selects the Hyper-DSNet, ADKNet, MSDDN, and LDPNet in the prior art as comparative methods to compare with the method proposed in the present application, and the results are as shown in Figure 8 .

[0148] In order to quantitatively evaluate the image fusion results, the present application embodiment introduces the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM), the spectral angle similarity (SAM), the average global error (ERGAS), and the image overall quality evaluation index (Q) as evaluation indexes, and the quantitative comparison results on the GaoFen-2 simulation dataset are as shown in Table 1:

[0149] Table 1

[0150] Method name PSNR↑ SSIM↑ ERGAS↓ Q↑ Hyper-DSNet 36.307 0.942 1.480 0.959 ADKNet 38.005 0.972 1.170 0.970 MSDDN 40.650 0.983 0.800 0.979 LDPNet 34.165 0.942 1.709 0.917 The present method 43.714 0.993 0.554 0.990

[0151] Further, the quantitative comparison results on the QuickBird real dataset are as shown in Table 2:

[0152] Table 2

[0153]

[0154]

[0155] According to the quantitative index results, the image fusion results obtained by the method proposed in the application are better than those of the existing methods on both the simulation data set and the real data set, and the high-quality high-resolution multispectral image fusion results can be generated on multiple satellite data.

[0156] Therefore, the application proposes a new universal multispectral image and panchromatic image fusion network based on frequency domain guidance. By introducing the frequency-guided image fusion mechanism, the network can be adaptively adjusted according to the different blur degrees of the input multispectral image. Meanwhile, the ConvNext module is introduced to improve the global modeling capability of the network for the input image. The spectral constraint subnetwork can adaptively estimate the multispectral image blur degradation, and use the estimated results to construct a spectral fidelity loss function to improve the spectral fidelity of the model output results. The method only needs to be trained on a single satellite, and can be effectively generalized to multiple different satellite data. A large number of simulation and actual experimental results prove the effectiveness and practicability of the application.

[0157] According to the universal multispectral and panchromatic image fusion method based on frequency domain guidance according to the embodiments of the application, the frequency domain guided multispectral and panchromatic image fusion network is constructed, the loss function of the spectral constraint subnetwork and the loss function of the universal image fusion subnetwork are used to train the frequency domain guided multispectral and panchromatic image fusion network, and the multispectral and panchromatic image fusion network after training is used to fuse the multispectral images and panchromatic images of multiple satellites. Therefore, the problems existing in the prior art, such as the fixed blur kernel degradation process not consistent with the real scene, lack of adaptability, and difficulty in realizing the universal fusion task in the real scene, are solved, and the network can be adaptively adjusted according to the different blur degrees of the input multispectral image.

[0158] Secondly, the universal multispectral and panchromatic image fusion system based on frequency domain guidance according to the embodiments of the application is described with reference to the accompanying drawings.

[0159] Figure 9 is a block schematic diagram of the universal multispectral and panchromatic image fusion system based on frequency domain guidance according to the embodiments of the application.

[0160] As shown in Figure 9 , the universal multispectral and panchromatic image fusion system 10 based on frequency domain guidance includes a first construction module 100, a second construction module 200, a third construction module 300, a fourth construction module 400, and a fusion module 500.

[0161] The first construction module 100 is configured to construct a spectral constraint subnetwork, wherein the spectral constraint subnetwork is configured to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result; the second construction module 200 is configured to construct a spatial detail injection module based on a frequency domain guide, wherein the spatial detail injection module is configured to inject spatial details of the panchromatic image into a fusion result; the third construction module 300 is configured to construct a general image fusion subnetwork, wherein the general image fusion subnetwork is configured to extract spatial details of to-be-fused images by using the spatial detail injection module; the fourth construction module 400 is configured to construct a frequency domain guided general multispectral and panchromatic image fusion network according to the spectral constraint subnetwork and the general image fusion subnetwork; and the fusion module 500 is configured to train the frequency domain guided multispectral and panchromatic image fusion network by using a loss function of the spectral constraint subnetwork and a loss function of the general image fusion subnetwork, and fuse multispectral images and panchromatic images of a plurality of satellites according to the trained multispectral and panchromatic image fusion network.

[0162] Further, in some embodiments, the spatial detail injection module includes a frequency domain spatial detail estimation module for generating detail injection weights of the panchromatic image, an Edge Detection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted high-frequency details of the panchromatic image.

[0163] Further, in some embodiments, the general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and the third construction module 300 is configured to: splice the up-sampled low-resolution multispectral image and the panchromatic image in the channel dimension, extract features of to-be-fused images at a plurality of resolution scales by using the U-shaped network architecture with the encoder-decoder, multiply the output feature result of the decoder with the to-be-injected spatial details generated by the spatial detail injection module, and connect the up-sampled low-resolution multispectral image to an output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

[0164] Further, in some embodiments, the encoder adopts a first preset number of ConvNext module cascaded structures, and the decoder adopts a second preset number of ConvNext module cascaded structures, wherein the first preset number is greater than the second preset number.

[0165] Further, in some embodiments, the loss function of the spectral constraint subnetwork is as follows:

[0166]

[0167] Y′=((X*C)↓spatial R)↓ spectral ;

[0168] Z' = ((X * R)↓ spectral C)↓ spatial ;

[0169] wherein, is a loss function of the spectral constraint subnetwork, Y' is a result of spectral degradation of the low-resolution multispectral image, Z' is a result of spatial degradation of the panchromatic image, X is the high-resolution multispectral image, C is blur in the spatial dimension, ↓ spatial is a downsampling operation in the spatial dimension, and R is a weight of each channel, spectral is a downsampling operation in the spectral dimension;

[0170] The loss function of the general image fusion subnetwork includes a spectral constraint term and a mean absolute error loss function;

[0171] wherein, the spectral constraint term is:

[0172]

[0173] wherein, is the spectral constraint term, is a fused high-resolution multispectral image, C is blur in the spatial dimension, ↓ spatial is a downsampling operation in the spatial dimension, and Y is the low-resolution multispectral image;

[0174] The mean absolute error loss function is:

[0175]

[0176] wherein, is the mean absolute error loss function, X is the high-resolution multispectral image, is the fused high-resolution multispectral image;

[0177] The loss function of the general image fusion subnetwork is:

[0178]

[0179] wherein, is the loss function of the general image fusion subnetwork, is the spectral constraint term, is the mean absolute error loss function.

[0180] It should be noted that the foregoing explanation and description of the embodiment of the general multispectral and panchromatic image fusion method based on frequency domain guidance also applies to the embodiment of the general multispectral and panchromatic image fusion system based on frequency domain guidance, which will not be described here.

[0181] The general multispectral and panchromatic image fusion system based on frequency domain guidance according to the embodiment of the application constructs a multispectral and panchromatic image fusion network guided by frequency domain, trains the multispectral and panchromatic image fusion network guided by frequency domain by using a loss function of a spectral constraint subnetwork and a loss function of a general image fusion subnetwork, and fuses multispectral images and panchromatic images of multiple satellites according to the multispectral and panchromatic image fusion network after training. Thus, the problems existing in the prior art, such as that the fixed blur kernel degradation process does not conform to the real scene, lacks adaptability, and it is difficult to realize the general fusion task in the real scene, are solved, and the adaptive adjustment of the network can be realized according to different blur degrees of the input multispectral images.

[0182] Figure 10 The electronic device provided by the embodiment of the application is shown in the structural schematic diagram. The electronic device can include:

[0183] The memory 1001, the processor 1002, and the computer program stored in the memory 1001 and executable on the processor 1002.

[0184] The processor 1002 implements the general multispectral and panchromatic image fusion method based on frequency domain guidance provided in the above embodiment when executing the program.

[0185] Further, the electronic device further includes:

[0186] The communication interface 1003 is used for communication between the memory 1001 and the processor 1002.

[0187] The memory 1001 is used for storing the computer program executable on the processor 1002.

[0188] The memory 1001 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0189] If the memory 1001, the processor 1002 and the communication interface 1003 are implemented independently, the communication interface 1003, the memory 1001 and the processor 1002 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 10 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0190] Optionally, in a specific implementation, if the memory 1001, the processor 1002 and the communication interface 1003 are integrated on a chip, the memory 1001, the processor 1002 and the communication interface 1003 can complete communication between each other through an internal interface.

[0191] The processor 1002 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0192] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the frequency domain guided general multispectral and panchromatic image fusion method as above.

[0193] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0194] In addition, the terms "first", "second", etc. are used only for the purpose of description, and should not be understood as indicating or implying relative importance or a specific number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited.

[0195] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A frequency domain guide-based general multispectral and panchromatic image fusion method, characterized in that, The method comprises the following steps: constructing a spectral constraint subnetwork, wherein the spectral constraint subnetwork is used to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result; constructing a spatial detail injection module based on a frequency domain guide, wherein the spatial detail injection module is used to inject spatial details of the panchromatic image into a fusion result; constructing a general image fusion subnetwork, wherein the general image fusion subnetwork is used to extract spatial details of to-be-fused images by using the spatial detail injection module; constructing a frequency domain guide general multispectral and panchromatic image fusion network according to the spectral constraint subnetwork and the general image fusion subnetwork; training the frequency domain guide multispectral and panchromatic image fusion network by using a loss function of the spectral constraint subnetwork and a loss function of the general image fusion subnetwork, and fusing multispectral images and panchromatic images of multiple satellites according to the trained multispectral and panchromatic image fusion network; wherein the spatial detail injection module based on the frequency domain guide comprises a frequency domain spatial detail estimation module for a low-resolution multispectral image, an Edge Detection module for estimating high-frequency details of a panchromatic image, and a Spatial Extraction module for integrating and injecting weighted high-frequency details of the panchromatic image; The frequency domain space detail estimation module obtains frequency domain information of the low-resolution multispectral image by first passing the low-resolution multispectral image through a Fourier transform, and the module passes the low-resolution multispectral image through 1 and 1 layer to generate an adaptive weight map, and then obtains the final panchromatic image space detail injection weight through average pooling. the Spatial Extraction module comprises a convolution layer and a Relu activation function, can nonlinearly extract high-frequency spatial features of the panchromatic image, multiply the obtained spatial detail injection weight with the high-frequency spatial details of the panchromatic image, and obtain the final injected spatial features through the Spatial Extraction module.

2. The method of claim 1, wherein, the general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and the extraction of the spatial details of the to-be-fused images by using the spatial detail injection module comprises: splicing the up-sampled low-resolution multispectral image and the panchromatic image in the channel dimension, and extracting features of the to-be-fused images at multiple resolution scales by using the U-shaped network architecture with the encoder-decoder; multiplying the output feature result of the decoder with the to-be-injected spatial details generated by the spatial detail injection module, and connecting the up-sampled low-resolution multispectral image to the output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

3. The method of claim 2, wherein, the encoder adopts a first preset number of ConvNext module cascade structures, and the decoder adopts a second preset number of ConvNext module cascade structures, wherein the first preset number is greater than the second preset number.

4. The method of claim 1, wherein, the loss function of the spectral constraint subnetwork is: ； ； ； wherein, is a loss function for the spectral constraint subnetwork, is a result of spectral degradation for a low-resolution multispectral image, is a result of spatial degradation for a panchromatic image, is a high-resolution multispectral image, is a blur for a spatial level, is a down-sampling operation for a spatial dimension, is a weight for each channel, is a down-scaling operation for a spectral level; the loss function of the general image fusion subnetwork comprises a spectral constraint term and a mean absolute error loss function; wherein the spectral constraint term is: ； wherein, is a spectral constraint term, is a resulting high-resolution multispectral image, is a spatial blur, is a downsampling operation in spatial dimension, is a low-resolution multispectral image; the mean absolute error loss function is: ； wherein, is the mean absolute error loss function, is the high resolution multispectral image, is the fused high resolution multispectral image; The loss function of the general image fusion subnetwork is: ； wherein, is a loss function of the general image fusion subnetwork, is a spectral constraint term, is a mean absolute error loss function.

5. A frequency domain guide-based general multispectral and panchromatic image fusion system, characterized in that, It comprises: A first construction module for constructing a spectral constraint subnetwork, wherein the spectral constraint subnetwork is used to take a low-resolution multispectral image and a panchromatic image as input, estimate a blur degradation result of the low-resolution multispectral image, and construct a spectral fidelity term of a loss function according to the blur degradation result; A second construction module for constructing a spatial detail injection module based on frequency domain guidance, wherein the spatial detail injection module is used to inject spatial details of the panchromatic image into a fusion result; A third construction module for constructing a general image fusion subnetwork, wherein the general image fusion subnetwork is used to extract spatial details of to-be-fused images by using the spatial detail injection module; A fourth construction module for constructing a frequency domain guided general multispectral and panchromatic image fusion network according to the spectral constraint subnetwork and the general image fusion subnetwork; A fusion module for training the frequency domain guided multispectral and panchromatic image fusion network by using the loss function of the spectral constraint subnetwork and the loss function of the general image fusion subnetwork, and fusing multispectral images and panchromatic images of multiple satellites according to the trained multispectral and panchromatic image fusion network; Wherein, the spatial detail injection module based on frequency domain guidance comprises a frequency domain spatial detail estimation module for low-resolution multispectral images, an Edge Detection module for estimating high-frequency details of the panchromatic image, and a Spatial Extraction module for integrating and injecting the weighted high-frequency details of the panchromatic image; The frequency domain space detail estimation module obtains frequency domain information of the low-resolution multispectral image by first passing the low-resolution multispectral image through a Fourier transform, and the module passes the low-resolution multispectral image through 1 and 1 layer to generate an adaptive weight map, and then obtains the final panchromatic image space detail injection weight through average pooling. The Spatial Extraction module contains a convolution layer and a Relu activation function, which can nonlinearly extract high-frequency spatial features of the panchromatic image, multiply the obtained spatial detail injection weight with the high-frequency spatial details of the panchromatic image, and obtain the final injected spatial features through the Spatial Extraction module.

6. The system of claim 5, wherein, The general image fusion subnetwork adopts a U-shaped network architecture with an encoder-decoder, and the third construction module is used to: Splice the up-sampled low-resolution multispectral image and the panchromatic image in the channel dimension, and extract features of the to-be-fused images at multiple resolution scales by using the U-shaped network architecture with the encoder-decoder; Multiply the output feature result of the decoder with the to-be-injected spatial details generated by the spatial detail injection module, and connect the up-sampled low-resolution multispectral image to the output end of the general image fusion subnetwork in a residual connection manner to obtain a fused high-resolution multispectral image.

7. An electronic device, comprising: It comprises: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the general multispectral and panchromatic image fusion method based on frequency domain guidance according to any one of claims 1-4.

8. A computer storage medium having stored thereon a computer program, characterized in that The program is executed by the processor for implementing the general multispectral and panchromatic image fusion method based on frequency domain guidance according to any one of claims 1-4.

Images