A multispectral and panchromatic image fusion method based on cross prediction

Abstract

The present invention discloses a multispectral and panchromatic image fusion method based on cross-prediction, including a cross-prediction proxy task and an unsupervised fusion module. The cross-prediction proxy task includes two branches, using a panchromatic image P to predict a multispectral image M (P2M), and using a multispectral image M to predict a panchromatic image P (M2P), both of which are trained in a self-supervised manner and implemented using a conditional diffusion model, including a forward denoising process and a reverse denoising process. The two trained noise prediction networks have the ability to extract spatial features and spectral features, respectively. The fusion module includes an attention module and a reconstruction module, and uses an unsupervised loss function to optimize network parameters until convergence. The present invention can extract the rich spatial detail features of panchromatic images and the spectral features of multispectral images, and generate clearer and more accurate multispectral and panchromatic image fusion results.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to an image fusion method. Background Art

[0002] Due to the hardware characteristics of satellite sensors and the limitation of signal-to-noise ratio, there is a certain trade-off between spatial resolution and spectral resolution in optical remote sensing images. Multispectral (MS) images have high spectral resolution but low spatial resolution; in contrast, panchromatic (PAN) images have relatively high spatial resolution, but they have only one band and low spectral resolution. In order to obtain remote sensing images with both high spatial resolution and high spectral resolution, image fusion is often used to fuse the spectral information of multispectral images and the spatial details of panchromatic images. Traditional multispectral and panchromatic image fusion methods attempt to separate spectral information and spatial information, and then fuse the spectral information of multispectral images with the spatial details of panchromatic images. Due to the limitation of model feature extraction capabilities, spectral information and spatial details are difficult to be effectively separated, which restricts the fusion effect of such methods.

[0003] Methods based on deep learning have attracted extensive attention in the fusion of multispectral and panchromatic images due to their powerful nonlinear modeling capabilities. However, due to the lack of high-resolution multispectral images as reference images in practice, most methods first construct pairs of images to be fused and reference images for supervised fusion, that is, panchromatic images and multispectral images are downsampled based on the Wald protocol as input, and then supervised learning is performed using the original multispectral images as reference information. The optimized network is directly applied to the fusion task of multispectral images and panchromatic images with original resolution. Due to the resolution difference between the image used to train the network and the original image to be fused, this type of supervised fusion model has poor generalization effect in practical applications. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a multispectral and panchromatic image fusion method based on cross prediction, including a cross prediction proxy task and an unsupervised fusion module. The cross prediction proxy task includes two branches, using the panchromatic image P to predict the multispectral image M (P2M), and using the multispectral image M to predict the panchromatic image P (M2P), both of which are trained in a self-supervised manner and implemented using a conditional diffusion model, including a forward denoising process and a reverse denoising process. The two trained noise prediction networks have the ability to extract spatial features and spectral features respectively. After freezing their parameters, the features extracted by the encoder of the noise prediction network are sent to the fusion module. The fusion module includes an attention module and a reconstruction module, and the network parameters are optimized using an unsupervised loss function until convergence. When the multispectral and panchromatic images are fused in the validation set or the test set, the noise prediction network of the P2M and M2P branches is first used to extract spatial and spectral features, and then the features are input into the fusion module to reconstruct the target image. The present invention can extract the rich spatial detail features of the panchromatic image and the spectral features of the multispectral image, and generate a clearer and more accurate multispectral and panchromatic image fusion result.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Dataset preparation;

[0007] Image blocks are cut from left to right and from top to bottom for paired, registered large-scale remote sensing images MS and PAN, and these image blocks are divided into training set, validation set and test set;

[0008] Normalize the training set, validation set, and test set;

[0009] Step 2: Construct a cross-prediction agent task;

[0010] The cross-prediction proxy task includes a P2M process and an M2P process, which are modeled using two conditional diffusion models respectively. The panchromatic image P and the multispectral image M are used as conditional information to reconstruct another image. The network learns the mapping relationship between the two images, thereby realizing cross-prediction.

[0011] Step 2-1: Forward noise addition process;

[0012] The multispectral image M and the panchromatic image P have a spatial resolution difference of r times. To maintain resolution consistency, the cross-prediction proxy task uses the multispectral image with r times upsampling. Or full-color image P0, a series of images are obtained by gradually adding noise based on the Markov chain process or {P0,P1,...,P T}, where T is the total step length of noise addition; the noise addition process is defined as follows:

[0013]

[0014] The noise scale α t Use predefined fixed parameters with a value range of (0,1) and use the cosine noise strategy to initialize α t ; The noisy image obtained by formula (1) and formula (2) obeys Gaussian distribution. If the total step length T of noisy is large enough, and P T Will follow a standard Gaussian distribution I represents a matrix with all values ​​1; represents Gaussian distribution;

[0015] Step 2-2: Reverse denoising process;

[0016] The reverse denoising process includes a P2M branch and an M2P branch. Both branches start from an image that obeys a Gaussian noise distribution and gradually remove the noise information contained in the image to reconstruct a multispectral image. Or full-color image P0; Specifically, the P2M branch uses the full-color image as conditional information to reconstruct the corresponding multispectral image; the M2P branch uses the upsampled multispectral image as conditional prior information to generate the corresponding full-color image; the noise prediction network is used in the denoising process, and the noise prediction network predicts the noise information contained in the image;

[0017] Specifically, for the P2M branch, the noise prediction network uses the full-color image P0 and the noisy multispectral image P2 at each time step. With the current time t as input, approximation Add noise to Noise information:

[0018] in, is the noise information predicted by the network at time t, represents the learnable parameters in the network, ∈ t1 express to Gaussian noise added; noise prediction network of P2M branch Parameter optimization is performed under formula (3-1) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence Generate a noisy multispectral image at the previous moment The mean and variance of are as follows:

[0019]

[0020]

[0021] in Obey Gaussian distribution; Based on formula (3-2) and formula (4), the upsampled multispectral image is reconstructed through Markov chain iteration with a step length of the total noise step length T The reconstruction process starts with a Gaussian noise and only uses the panchromatic image P0 as conditional prior information, so this process is called the P2M branch, that is, the panchromatic image is used to reconstruct the corresponding upsampled multispectral image; the P2M branch maps the spatial attributes of the panchromatic image to the corresponding multispectral image, so the network has the ability to extract features from the panchromatic image P;

[0022] In contrast, the M2P process uses an upsampled multispectral image As conditional prior information to generate the corresponding full-color image; the noise prediction network is Noised full-color image P t With time t as input, to approximate P0 to P t The noise information added, the optimization objectives are as follows:

[0023]

[0024] in, is the noise information predicted by the network at time t, θ represents the learnable parameters in the network, ∈ t2 Indicates P0 to P t Gaussian noise added; noise prediction network ∈ of M2P branch θ Parameter optimization is performed under formula (5) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence Generate the noisy full-color image P of the previous moment t-1 The mean and variance of :

[0025]

[0026]

[0027] Similarly, P t-1 Obey Gaussian distribution; the reverse denoising process is based on formula (6) and formula (7) to reconstruct the full-color image through Markov chain iteration with a step length of the total step length T of the noise addition The reconstruction process of the panchromatic image starts with a Gaussian noise and uses only the upsampled multispectral image It is conditional prior information, so this process is called M2P branch, that is, the corresponding panchromatic image is reconstructed using the multispectral image;

[0028] The two branches of the cross-prediction agent task are trained using a self-supervised method; the network model is trained using the training set, and the validation set is used during the training process to observe the model performance and obtain the network model with the best reconstruction performance.

[0029] Step 3: Build a noise prediction network;

[0030] The noise prediction network adopts a U-shaped network architecture, which includes four parts: time step module, downsampling module, upsampling module and prediction module; the input information received by the network includes the time step t, the noise image P at that moment t or and condition information or P0, P t and Corresponding to the M2P branch, And P0 corresponds to the P2M branch; at the input end, the conditional information and the noisy image are connected in the channel dimension and input into a convolution layer for unified processing;

[0031] The specific construction process of each part is as follows:

[0032] Step 3-1: Construct a time step module;

[0033] The noise prediction network is related to the time step. The input integer time step t~{1,...,T} is encoded as a one-dimensional vector, which is then passed through a Linear linear change layer. The vector will be passed into all upsampling, downsampling and prediction modules of the U-shaped network structure. Before being passed into each module, the vector will pass through another Linear linear change layer to convert its length into the number of input channels of the current module, so as to be summed with the input of the current module.

[0034] Step 3-2: Construct downsampling module;

[0035] The downsampling module is a structure of a group normalization GN layer, a leaklyRelu activation function, and a convolutional layer in sequential cascade; the downsampling operation of the convolutional layer can be achieved by adjusting the step size, downsampling the feature map by 2 times, and the number of output channels is doubled relative to the number of input channels;

[0036] Step 3-3: Build upsampling module;

[0037] The upsampling module is a structure of sequentially cascading a group normalized GN layer, a leaklyRelu activation function, an upsampling layer, a skip connection, and a convolutional layer with a step size of 1; thereby upsampling the feature map by a factor of 2, and the number of output channels is halved relative to the number of input channels; the convolutional layer of the upsampling module uses a skip connection to receive the output of the downsampling module of the same size; the upsampling layer uses a bilinear interpolation algorithm to achieve a 2-fold spatial upsampling;

[0038] Step 3-4: Build prediction module;

[0039] The prediction module is a structure of sequentially cascading a group normalized GN layer, a leaklyRelu activation function, a skip connection, and a convolutional layer with a step size of 1, keeping the size of the feature map unchanged; the convolutional layer of the prediction module uses a skip connection to receive the output of the first convolutional layer and the network input; specifically, the network input is P0 and In the M2P branch and P t ;

[0040] Step 4: Build fusion module;

[0041] After training, the parameters of the noise prediction network are fixed. The noise prediction network encoders of P2M and M2P are used to extract spatial and spectral features and pass them into the fusion module. The encoder part of the noise prediction network includes a convolutional layer and three downsampling modules to extract multi-scale features.

[0042] The fusion module consists of an attention module and a reconstruction module; the attention module uses three attention layers and three upsampling layers to perform difference operations and feature fusion on the multi-scale features extracted by the encoder part of the P2M and M2P branches; specifically, the features are spatially upsampled twice after passing through an attention layer, and then added to the features of the next scale, and then pass through an attention layer again and perform spatial upsampling twice, and this process is repeated until it is summed with the features with the largest shape to obtain the fused features and input them into the reconstruction module; the output of the reconstruction module is summed with the upsampled multispectral image to finally obtain the fusion result.

[0043] Step 5: Fusion module training;

[0044] Use unsupervised loss function to include no reference indicator items A spatial term and a spectral term The details are as follows:

[0045]

[0046]

[0047]

[0048]

[0049] Among them, GS(.) represents Gaussian blur operation; SSIM(.) represents the structural similarity index, λ is a constant, FMS represents the fused image output by the fusion module, and FMS r↓ It represents the result of spatially downsampling the fused image by a factor of r.

[0050] Preferably, when the noise prediction network encoder performs feature extraction, the noise-added image is input Select the noise adding time t as 50;

[0051] Preferably, the normalization process is to divide the pixel values ​​of all input images by 2047.0.

[0052] Preferably, the upsampled multispectral image M r↑ It means upsampling the MS image block to the spatial resolution of the PAN image block by bicubic interpolation, and the upsampling factor r is 4.

[0053] Preferably, the PAN image block size of the training set and validation set is 256×256, and the MS image block size is 64×64×4; the PAN image block size of the test set is 1024×1024, and the MS image block size is 256×256×4; the ratio of the data volume of the training set, validation set, and test set is 8:1:1.

[0054] Preferably, T is set to 1000 and λ is set to 0.0001.

[0055] The beneficial effects of the present invention are as follows:

[0056] The present invention proposes a multispectral and panchromatic image fusion method based on cross-prediction, which comprises two parts: a cross-prediction proxy task and an unsupervised fusion module. The cross-prediction proxy task is implemented using a conditional diffusion model, which can extract the rich spatial detail features of the panchromatic image and the spectral features of the multispectral image. The fusion module is optimized using an unsupervised loss function, which avoids the problem of inconsistent resolution between the training set and the test set, and generates clearer and more accurate multispectral and panchromatic image fusion results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the method of the present invention.

[0058] Figure 2 It is a schematic diagram of the cross prediction task of the method of the present invention.

[0059] Figure 3 It is a noise prediction network structure diagram constructed by the method of the present invention.

[0060] Figure 4 It is a schematic diagram of the fusion module constructed by the method of the present invention. DETAILED DESCRIPTION

[0061] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0062] The present invention designs a cross-prediction proxy task under a self-supervised framework, obtains a spectral feature extractor and a spatial detail extractor through the proxy task, and then fuses the multispectral and panchromatic images in an unsupervised manner.

[0063] The present invention provides a multi-spectral and panchromatic image fusion method based on cross prediction, which is implemented by a cross prediction proxy task and a fusion module. The cross prediction proxy task includes two branches: panchromatic image P predicts multi-spectral image M (P2M) and multi-spectral image M predicts panchromatic image P (M2P), which is implemented by a conditional diffusion model. After the trained noise prediction network parameters are fixed, the features extracted by the noise prediction network encoders of P2M and M2P are sent to the fusion module. The fusion module includes an attention module and a reconstruction module. The attention module performs difference operations and attention calculations on the multi-scale features extracted by the encoder parts of the P2M and M2P branches, respectively, for weighted fusion of the features. The reconstruction module uses the weighted fused features to output the final multi-spectral and panchromatic image fusion results. The noise prediction network adopts a U-type network architecture, and optimizes the model parameters by a heavily weighted variational lower bound loss function. The noise prediction networks of the P2M and M2P branches have spatial feature extraction capabilities and spectral feature extraction capabilities, respectively.

[0064] When multispectral and panchromatic image fusion is performed in the validation set or test set, the noise prediction network of the P2M and M2P branches is first used to extract spatial and spectral features, and then the features are input into the fusion module to reconstruct the target image. The cross-prediction proxy task of the present invention can extract the rich spatial detail features of the panchromatic image and the spectral features of the multispectral image, which are used to guide the fusion module to generate clearer and more accurate multispectral and panchromatic image fusion results.

[0065] A multispectral and panchromatic image fusion method based on cross prediction includes the following steps:

[0066] Step 1: Dataset preparation;

[0067] The data comes from QuickBird (QB), WorldView-4 (WV-4), and WorldView-2 (WV-2) satellite sensors. For QB data, the spatial resolution of PAN is 0.6 meters, the spatial resolution of MS is 2.4 meters, and MS contains 4 spectral bands: blue, green, red, and near-infrared bands. For WV-4 data, the spatial resolution of PAN is 0.3 meters, the spatial resolution of MS is 1.2 meters, and MS contains 4 spectral bands: blue, green, red, and near-infrared bands. For WV-2 data, the spatial resolution of PAN is 0.5 meters, the spatial resolution of MS is 2 meters, and MS contains 8 spectral bands: cyan, yellow, infrared, blue, green, red, near-infrared 1 band, and near-infrared 2 band. The spatial resolution ratio between MS and PAN images in the three datasets is 4.

[0068] The image blocks at the corresponding positions of the large-scale paired and registered MS images and PAN images are cut from left to right and from top to bottom without overlap, and the cut image blocks are divided into three parts, which are used as training set, validation set and test set respectively; the image block size of the training set and validation set is 256×256 (PAN) / 64×64×4 (MS), and the image block size of the test set is 1024×1024 (PAN) / 256×256×4 (MS), and the ratio of training, validation and test data is 8:1:1. PAN image blocks are denoted as P, and MS image blocks are denoted as M.

[0069] Step 2: Construction of cross-prediction agent task;

[0070] The cross-prediction proxy task includes a P2M process and an M2P process. P2M and M2P are modeled as two conditional diffusion models, in which the panchromatic image P and the multispectral image M are used as conditional information to reconstruct the other image. The specific process is as follows: Figure 2 As shown;

[0071] Step 2-1: Forward noise addition process;

[0072] The multispectral image M and the panchromatic image P have a spatial resolution difference of r times. To maintain resolution consistency, the cross-prediction proxy task uses the multispectral image with r times upsampling. Or full-color image P0, the forward noise process is based on the Markov chain process to gradually add noise to obtain a series of images or {P0,P1,...,P T}, where the total step length T of noise addition is set to 1000; the noise addition process is defined as follows:

[0073]

[0074]

[0075] The noise scale α t Use predefined fixed parameters with a value range of (0,1) and cosine noise strategy to initialize α t ; The cosine noise initialization strategy is described as in The value of s is 0.008. The noisy image obtained by formula (1) and formula (2) obeys the Gaussian distribution. If the total step length T of the noisy image is large enough, and P T Will follow a standard Gaussian distribution

[0076] Step 2-2: Reverse denoising process

[0077] The reverse denoising process includes a P2M branch and an M2P branch, both of which start from an image that obeys a Gaussian noise distribution and gradually remove the noise information contained in the image to reconstruct a multispectral image. or full-color image P0;

[0078] Specifically, for the P2M branch, the noise prediction network uses the full-color image P0 and the noisy multispectral image P2 at each time step. With the current time t as input, approximation Add noise to Noise information:

[0079]

[0080] in, is the noise information predicted by the network at time t, represents the learnable parameters in the network, ∈ t1 express to Gaussian noise added; noise prediction network of P2M branch Parameter optimization is performed under formula (3-1) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence

[0081] Can generate a noisy multispectral image of the previous moment The mean and variance of are as follows:

[0082]

[0083]

[0084] in Obey Gaussian distribution; Based on formula (3-2) and formula (4), the upsampled multispectral image is reconstructed iteratively through a Markov chain with a step length of T for the total step length of noise addition The reconstruction process starts from a Gaussian noise and only uses the panchromatic image P0 as conditional prior information, so this process is called the P2M branch, that is, the panchromatic image is used to reconstruct the corresponding upsampled multispectral image; the P2M branch maps the spatial attributes of the panchromatic image to the corresponding multispectral image, so the network has the ability to extract features of the panchromatic image P;

[0085] In contrast, the M2P process uses an upsampled multispectral image As conditional prior information to generate the corresponding full-color image; the noise prediction network is Noised full-color image P t With time t as input, to approximate P0 to P t The noise information added, the optimization objectives are as follows:

[0086]

[0087] in, is the noise information predicted by the network at time t, θ represents the learnable parameters in the network, ∈ t2 Indicates P0 to P t Gaussian noise added; noise prediction network ∈ of M2P branch θ Parameter optimization is performed under formula (5) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence The noisy full-color image P of the previous moment can be generated t-1 The mean and variance of :

[0088]

[0089]

[0090] Similarly, P t-1 Obey Gaussian distribution; the reverse denoising process is based on formula (6) and formula (7) through the Markov chain iterative reconstruction of the full-color image with a step length of T for the total step length of noise addition. The reconstruction process of the panchromatic image starts with a Gaussian noise and uses only the upsampled multispectral image It is conditional prior information, so this process is called M2P branch, that is, the corresponding panchromatic image is reconstructed using the multispectral image. The network will learn how to map the spectral attributes to the corresponding spatial attributes. The trained noise prediction network has the ability to extract spectral features of multispectral images.

[0091] The noise prediction networks of the two branches are trained using the same process; specifically, the Adam optimizer is used to optimize the model with an initial learning rate of 0.0003, and the training is stopped when the loss function of the model converges.

[0092] Step 3: Noise prediction network construction

[0093] The noise prediction network adopts a U-shaped network architecture. Figure 3 As shown in Figure 1, it includes four parts: time step module, downsampling module, upsampling module and prediction module. The input information received by the network includes time step t, the noisy image P at that moment t or Condition Information or P0, corresponding to the M2P branch and the P2M branch respectively;

[0094] At the input end, the conditional information and the noisy image are concatenated in the channel dimension and then input into the convolutional layer for unified processing. The window size can be 3×3, the step size is 1, and the output channel is 32;

[0095] The specific construction process of each part is as follows:

[0096] Step 3-1: Construct a time step module;

[0097] The noise prediction network is related to the time step. The input integer time step t~{1,...,T} is encoded as a one-dimensional vector, and then passes through the Linear linear change layer. The vector will be passed into all upsampling, downsampling and fusion modules of the U-shaped network structure; before being passed into each module, the vector will pass through a Linear linear change layer to convert its length into the number of input channels of the current module, so as to be summed with the input of the current module.

[0098] Step 3-2: Build a downsampling module

[0099] The downsampling module is a structure of a group normalization GN layer, a leaklyRelu activation function, and a convolutional layer in sequential cascade; the convolutional layer window size is 3×3, the padding is 2, and the step size is 1, so that the feature map is downsampled by 2 times, and the number of output channels is doubled relative to the number of input channels;

[0100] Step 3-3: Build upsampling module

[0101] The upsampling module is a structure of a group normalization GN layer, a leaklyRelu activation function, an upsampling layer, a skip connection, and a convolutional layer in sequential cascade; the convolutional layer window size is 3×3, the padding is 1, and the step size is 1, so that the feature map is upsampled by 2 times, and the number of output channels is halved relative to the number of input channels; the convolutional layer of the upsampling module uses a skip connection to receive the output of the downsampling module of the same size; the upsampling layer uses a bilinear interpolation algorithm to achieve 2 times spatial upsampling;

[0102] Step 3-4: Build the prediction module

[0103] The prediction module is a structure of a group normalized GN layer, a leaklyRelu activation function, a skip connection, and a sequential cascade of convolutional layers; the convolutional layer window size is 3×3, the padding is 1, and the step size is 1, so as to keep the feature map size unchanged; the convolutional layer of the prediction module uses a skip connection to receive the output of the first convolutional layer and the network input;

[0104] Step 4: Fusion module construction

[0105] After training, the parameters of the noise prediction network are fixed. The noise prediction network encoders of P2M and M2P are used to extract spatial and spectral features and pass them into the fusion module. The encoder part of the noise prediction network includes a convolutional layer and three downsampling modules to extract multi-scale features. The extracted feature shapes are: H×W×32, Where H and W represent the height and width of the full-color image respectively;

[0106] The fusion module consists of an attention module and a reconstruction module. The attention module uses three attention layers and three upsampling layers to perform difference operations and feature fusion on the multi-scale features extracted by the encoder part of the P2M and M2P branches. Specifically, the lower-scale features pass through an attention layer and are spatially upsampled twice, and then added to the features of the next scale, and then pass through an attention layer again and are spatially upsampled twice. This process is repeated until it is summed with the features with the largest shape, and the fused features are obtained and input into the reconstruction module. The output of the reconstruction module is summed with the upsampled multispectral image to finally obtain the fusion result.

[0107] Step 5: Fusion module training

[0108] The unsupervised loss function defined in the present invention includes a reference-free index term QNR, a spatial term and a spectral term, which are specifically defined as follows:

[0109]

[0110]

[0111]

[0112]

[0113] Among them, the value of λ is 0.0001, the reference-free index QNR uses the implementation method of the paper "Multispectral and panchromaticdata fusion assessment without reference"; GS represents the Gaussian blur operation, the Gaussian blur kernel window size used is 5, and the standard deviation is 2; SSIM represents the structural similarity index, and the sliding window size is set to 5;

[0114] Specifically, the Adam optimizer is used to optimize the fusion module with an initial learning rate of 0.0003, and the training is stopped when the loss function formula (8) of the model converges. The network model is trained using the training set, and the validation set is used during the training process to observe the model performance and obtain the best performing network model. Finally, the test set is used to test the effectiveness of the model. Specific embodiment:

[0116] (1) Dataset preparation

[0117] The panchromatic image and multispectral image with a width-to-height ratio of 4:1 and registered with each other are processed as follows:

[0118] ① Read the image and divide it into two parts, which are used as training data image and test data image respectively. The principle of division is to make the width of the two parts the same and the height ratio of 9:1. Both PAN and MS do this processing;

[0119] ② For the training data part, the image blocks of the corresponding positions of the matching PAN and MS training images are intercepted from left to right and from top to bottom, where the image block size of PAN is 256×256 and the image block size of MS is 64×64×4 (4 is the number of channels, which can be changed to 8 when the number of MS channels is 8). The test data part is constructed in a similar way, where the image block size of PAN is 1024×1024 and the image block size of MS is 256×256×4.

[0120] ③ Randomly divide 1 / 9 of the training data into validation set data.

[0121] So far, we have obtained the training set, validation set, and test set data. For QB, the training set contains 6943 pairs of images, the validation set contains 743 pairs of images, and the test set contains 156 pairs of images; for WV-4, the training set contains 7166 pairs of images, the validation set contains 772 pairs of images, and the test set contains 271 pairs of images; for WV-2, the training set contains 9641 pairs of images, the validation set contains 945 pairs of images, and the test set contains 136 pairs of images.

[0122] At this point, the dataset preparation step is completed.

[0123] (2) Construction of noise prediction network model

[0124] The P2M branch and the M2P branch each use a noise prediction network to achieve cross prediction; see the network structure diagram Figure 3 , the important parameters for building the network include:

[0125] ① Convolutional layers used in the entire network: the window size of all convolutional layers is 3×3; the convolutional layer stride of the downsampling module is 2 and the padding is 1; the convolutional layer stride of other modules is 1 and the padding is 1.

[0126] Figure 3 The number of output channels of the middle convolutional layer is 32, 64, 128, 256, 128, 64, 32, 32, and the target number of channels respectively; the jump connection occurs in the upsampling module and the fusion module, and the input information and the output of the downsampling module are passed to the second half of the network, connecting the features with the same resolution in the channel dimension;

[0127] ② The temporal coding layer uses the sine-cosine coding layer from the paper “Attention is all you need” to encode the time step t into a one-dimensional vector of length 32. This vector is converted into a vector of length 256 through the Linear layer. Before being added to the input of each module, the linear change layer is used to modify the number of channels to make the remaining input channels consistent.

[0128] ③The last layer output channel of the P2M branch network is 4, and the last layer output channel of the M2P branch network is 1;

[0129] (3) Construction of fusion module

[0130] Network structure diagram Figure 4 , the important parameters for building the network include:

[0131] ① The convolutional layers used in the entire network: the window size of all convolutional layers is 3*3, the padding is 1, the stride is 1, the output channels of the convolutional layers of the three groups of attention layers are 128, 64, and 32 respectively; the output channel of the convolutional layer of the reconstruction module is 32, the number of spectra of the multispectral image;

[0132] ② The scSE layer uses the structure of the document “Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks”. Preferably, the channel number squeezing factor is 2.

[0133] (4) Cross-prediction agent task training

[0134] The P2M branch and the M2P branch use exactly the same training parameter settings;

[0135] ① Input image: full color image P, size 256×256 and upsampled multispectral image M r↑ , size 256×256×4 (height×width×number of channels). Here the multispectral M is upsampled r↑The multi-spectrum of size 64×64×4 is obtained by upsampling 4 times through bilinear interpolation algorithm. The image input to the network is normalized, which means that the pixel value of the input image is divided by 2047.0.

[0136] ②Other related settings: Use Adam optimizer to update parameters. Set the number of training epochs to 200, the batch size to 32, and the initial learning rate to 0.0003. Use the validation set to test the network effect after every 50 epochs, and save the network parameters with the best effect.

[0137] ③ Conditions for stopping training: the network’s loss function reaches convergence.

[0138] (5) Fusion module training

[0139] ① Input features: full color image P and upsampled multispectral M r↑ First, through the Markov chain noise process until t = 50, we get The parameters of the two U-shaped network architecture models of P2M and M2P are fixed, and their encoder parts are used to extract multi-scale features respectively; for P2M, the encoder input is P, t three pieces of information, for M2P, the encoder input M r↑ , P 50 , t;

[0140] ② Other related settings: The Adam optimizer is used to update parameters in the PyTorch framework. The number of training epochs is set to 50, the batch size is set to 32, and the initial learning rate is set to 0.0003. The validation set is used to test the network effect after every 5 epochs, and the network parameters with the best effect are saved.

[0141] ③ Conditions for stopping training: the network’s loss function reaches convergence.

[0142] (6) Multispectral and panchromatic image fusion test

[0143] ① Input image: full-color image P, size 256×256 (height×width) and upsampled multispectral image M r↑ , size 256×256×4 (height×width×number of channels). Here the multispectral M is upsampled r↑ The multi-spectrum with size 64×64×4 is upsampled by 4 times using bilinear interpolation algorithm.

[0144] ② Load the best fusion network parameters saved in the training phase, the noise prediction network parameters of P2M, and the noise prediction network parameters of M2P, and select a pair of full-color image P and upsampled multi-spectral image M from the test set r↑ , add noise for 50 steps to get Input P2M noise prediction network, (M r↑ , P 50 , t) Input the noise prediction network of M2P, the two encoder parts extract multi-scale features and input them into the fusion network; the fusion network uses the multi-scale features to output the fusion result.

Claims

1. A multispectral and panchromatic image fusion method based on cross prediction, characterized in that: The steps include: Step 1: Dataset preparation; Image blocks are cut from left to right and from top to bottom for paired, registered large-scale remote sensing images MS and PAN, and these image blocks are divided into training set, validation set and test set; Normalize the training set, validation set, and test set; Step 2: Construct a cross-prediction agent task; The cross-prediction proxy task includes a P2M process and an M2P process, which are modeled using two conditional diffusion models respectively. The panchromatic image P and the multispectral image M are used as conditional information to reconstruct another image. The network learns the mapping relationship between the two images, thereby realizing cross-prediction. Step 2-1: Forward noise addition process; The multispectral image M and the panchromatic image P have a spatial resolution difference of r times. To maintain resolution consistency, the cross-prediction proxy task uses the multispectral image with r times upsampling. Or full-color image P0, a series of images are obtained by gradually adding noise based on the Markov chain process or {P0,P1,...,P T }, where T is the total step length of noise addition; the noise addition process is defined as follows: The noise scale α t Use predefined fixed parameters with a value range of (0,1) and use the cosine noise strategy to initialize α t ; The noisy image obtained by formula (1) and formula (2) obeys Gaussian distribution. If the total step length T of noisy is large enough, and P T Will follow a standard Gaussian distribution I represents a matrix with all values ​​1; represents Gaussian distribution; Step 2-2: Reverse denoising process; The reverse denoising process includes a P2M branch and an M2P branch. Both branches start from an image that obeys a Gaussian noise distribution and gradually remove the noise information contained in the image to reconstruct a multispectral image. Or full-color image P0; Specifically, the P2M branch uses the full-color image as conditional information to reconstruct the corresponding multispectral image; the M2P branch uses the upsampled multispectral image as conditional prior information to generate the corresponding full-color image; the noise prediction network is used in the denoising process, and the noise prediction network predicts the noise information contained in the image; Specifically, for the P2M branch, the noise prediction network uses the full-color image P0 and the noisy multispectral image P2 at each time step. With the current time t as input, approximation Add noise to Noise information: in, is the noise information predicted by the network at time t, represents the learnable parameters in the network, ∈ t1 express to Gaussian noise added; noise prediction network of P2M branch Parameter optimization is performed under formula (3-1) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence Generate a noisy multispectral image at the previous moment The mean and variance of are as follows: in Obey Gaussian distribution; Based on formula (3-2) and formula (4), the upsampled multispectral image is reconstructed through Markov chain iteration with a step length of the total noise step length T The reconstruction process starts with a Gaussian noise and only uses the panchromatic image P0 as conditional prior information, so this process is called the P2M branch, that is, the panchromatic image is used to reconstruct the corresponding upsampled multispectral image; The M2P process uses an upsampled multispectral image As conditional prior information to generate the corresponding full-color image; the noise prediction network is Noised full-color image P t With time t as input, to approximate P0 to P t The noise information added, the optimization objectives are as follows: in, is the noise information predicted by the network at time t, θ represents the learnable parameters in the network, ∈ t2 Indicates P0 to P t Gaussian noise added; noise prediction network ∈ of M2P branch θ Parameter optimization is performed under formula (5) until the loss function reaches a convergence state and training is stopped; the noise prediction network after convergence Generate the noisy full-color image P of the previous moment t-1 The mean and variance of : P t-1 Obey Gaussian distribution; the reverse denoising process is based on formula (6) and formula (7) to reconstruct the full-color image through Markov chain iteration with a step length of the total step length T of the noise addition The two branches of the cross-prediction proxy task are trained using self-supervision. The network model is trained using the training set, and the validation set is used during the training process to observe the model performance and obtain the network model with the best reconstruction performance. Step 3: Build a noise prediction network; The noise prediction network adopts a U-shaped network architecture, which includes four parts: time step module, downsampling module, upsampling module and prediction module; the input information received by the network includes the time step t, the noise image P at that moment t or and condition information or P0, P t and Corresponding to the M2P branch, And P0 corresponds to the P2M branch; at the input end, the conditional information and the noisy image are connected in the channel dimension and input into a convolution layer for unified processing; The specific construction process of each part is as follows: Step 3-1: Construct a time step module; The noise prediction network is related to the time step. The input integer time step t~{1,...,T} is encoded as a one-dimensional vector, which is then passed through a Linear linear change layer. The vector will be passed into all upsampling, downsampling and prediction modules of the U-shaped network structure. Before being passed into each module, the vector will pass through another Linear linear change layer to convert its length into the number of input channels of the current module, so as to be summed with the input of the current module. Step 3-2: Construct downsampling module; The downsampling module is a structure of a group normalization GN layer, a leaklyRelu activation function, and a convolutional layer in sequential cascade; the downsampling operation of the convolutional layer can be achieved by adjusting the step size, downsampling the feature map by 2 times, and doubling the number of output channels relative to the input channels; Step 3-3: Build upsampling module; The upsampling module is a structure of sequentially cascading a group normalized GN layer, a leaklyRelu activation function, an upsampling layer, a skip connection, and a convolutional layer with a step size of 1; thereby upsampling the feature map by a factor of 2, and the number of output channels is halved relative to the number of input channels; the convolutional layer of the upsampling module uses a skip connection to receive the output of the downsampling module of the same size; the upsampling layer uses a bilinear interpolation algorithm to achieve a 2-fold spatial upsampling; Step 3-4: Build prediction module; The prediction module is a structure of sequentially cascading a group normalized GN layer, a leaklyRelu activation function, a skip connection, and a convolutional layer with a step size of 1, keeping the size of the feature map unchanged; the convolutional layer of the prediction module uses a skip connection to receive the output of the first convolutional layer and the network input; specifically, the network input is P0 and In the M2P branch and P t ; Step 4: Build fusion module; After training, the parameters of the noise prediction network are fixed. The noise prediction network encoders of P2M and M2P are used to extract spatial and spectral features and pass them into the fusion module. The encoder part of the noise prediction network includes a convolutional layer and three downsampling modules to extract multi-scale features. The fusion module consists of an attention module and a reconstruction module; the attention module uses three attention layers and three upsampling layers to perform difference operations and feature fusion on the multi-scale features extracted by the encoder part of the P2M and M2P branches; specifically, the features are spatially upsampled twice after passing through an attention layer, and then added to the features of the next scale, and then passed through an attention layer and spatially upsampled twice again, and this process is repeated until it is summed with the features with the largest shape, and the fused features are obtained and input into the reconstruction module; the output of the reconstruction module is summed with the upsampled multispectral image to finally obtain the fusion result; Step 5: Fusion module training; Use unsupervised loss function to include no reference indicator items A spatial term and a spectral term The details are as follows: Among them, GS(.) represents Gaussian blur operation; SSIM(.) represents the structural similarity index, λ is a constant, FMS represents the fused image output by the fusion module, and FMS r↓ It represents the result of spatially downsampling the fused image by a factor of r.

2. The method for fusion of multispectral and panchromatic images based on cross prediction according to claim 1, characterized in that: The noise prediction network encoder inputs the noisy image when performing feature extraction. The noise adding time t is selected as 50.

3. The multispectral and panchromatic image fusion method based on cross prediction according to claim 1, characterized in that: The normalization process is to divide the pixel values ​​of all input images by 2047.

0.

4. The multispectral and panchromatic image fusion method based on cross prediction according to claim 1, characterized in that: The upsampled multispectral image M r↑ It means upsampling the MS image block to the spatial resolution of the PAN image block by bicubic interpolation, and the upsampling factor r is 4.

5. The method for fusion of multispectral and panchromatic images based on cross prediction according to claim 1, characterized in that: The PAN image block size of the training set and validation set is 256×256, and the MS image block size is 64×64×4; the PAN image block size of the test set is 1024×1024, and the MS image block size is 256×256×4; the ratio of the data volume of the training set, validation set, and test set is 8:1:

1.

6. The method for fusion of multispectral and panchromatic images based on cross prediction according to claim 1, characterized in that: The total step length T of noise addition is set to 1000, and the value of λ is 0.0001.

Images