A Strong Generalized Pan-sharpening Method Based on Text Modulation Diffusion Model

Abstract

The present invention discloses a strong generalization panchromatic sharpening method based on a text modulation diffusion model, which uses text prompts to provide information related to satellite data, modulates the parameters of the diffusion model, and uses a 3D convolutional neural network to build a model. The model consists of a forward diffusion process and a reverse denoising process, wherein the forward diffusion process gradually adds Gaussian noise to the target image through a Markov chain method, and the reverse denoising process gradually generates a target high-resolution multispectral image from noise data of a standard Gaussian distribution. The present invention designs a text-modulated 3D convolutional neural network, in which the text prompt generates text encoding through the text encoder of CLIP, which is multiplied with the convolution kernel of the 3D convolutional neural network in the channel dimension, generates adaptive network parameters for different satellite data, and processes multiple satellite data sets. The present invention can utilize multiple remote sensing data sets, thereby reducing spectral distortion while enhancing spatial detail information, and greatly enhancing the generalization ability of the model in unknown data sets.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a strong generalized full color sharpening method based on a text modulation diffusion model. Background Art

[0002] At present, most optical earth observation satellites, such as WorldView-4, QuickBird, WorldView-2, etc., can simultaneously capture remote sensing images with complementary characteristics through two sensors, namely, multispectral (MS) images with high spectral resolution and low spatial resolution and panchromatic (PAN) images with high spatial resolution and low spectral resolution, which provide sufficient data sources for pan-sharpening technology. Pan-sharpening is a technology that fuses multispectral and panchromatic images. Its essence is to use the fine spatial detail information in the panchromatic image to sharpen the multispectral image, thereby obtaining a multispectral image with high spatial resolution. It is beneficial to tasks such as change detection, target detection, and land classification. This technology can also be used to achieve image enhancement to improve image readability, so it plays an important role in real life.

[0003] The key to panchromatic sharpening technology is to use the detail information of panchromatic images to enhance the spatial resolution of multispectral images. Current research usually collects observation datasets of specific satellites and trains models on panchromatic-multispectral image pairs of specific satellites. However, they only perform well on the training datasets. When applied to unseen satellite data, they are even worse than traditional methods, and there is a problem of inconsistent number of spectral channels. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a strong generalization panchromatic sharpening method based on a text modulation diffusion model, which uses text prompts to provide information related to satellite data, modulates the parameters of the diffusion model, and uses a 3D convolutional neural network to build a model. The model can process satellite data sets with inconsistent numbers of spectral channels and obtain better generalization performance. The model consists of a forward diffusion process and a reverse denoising process, wherein the forward diffusion process gradually adds Gaussian noise to the target image through a Markov chain method, and the reverse denoising process gradually generates a target high-resolution multispectral image from the noise data of the standard Gaussian distribution. In the above reverse denoising process, the text prompt and the observed image pair (panchromatic image and multispectral image) are used as conditional information to guide the sampling of the 3D diffusion model. The present invention designs a text-modulated 3D convolutional neural network, wherein the text prompt generates a text encoding through the text encoder of CLIP, which is multiplied by the convolution kernel of the 3D convolutional neural network in the channel dimension, and adaptive network parameters are generated for different satellite data, so as to process multiple satellite data sets with a unified model. The present invention can make full use of multiple remote sensing data sets, thereby reducing spectral distortion while enhancing spatial detail information, and greatly enhancing the generalization ability of the model in unknown data sets.

[0005] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0006] Step 1: Dataset preparation;

[0007] Image blocks are cut from left to right and from top to bottom for paired, registered remote sensing multispectral MS images and panchromatic images PAN, and these image blocks are divided into training set, validation set and test set;

[0008] First, the training set, validation set, and test set are normalized; then the image blocks in the training set, validation set, and test set are processed according to the Wald protocol, and then the processed image blocks are used as the input of the model;

[0009] The original MS image block is used as the reference image;

[0010] Step 2: Forward noise addition process;

[0011] Set the total step length of noise addition to T. For any time step t~{1,...,T}, calculate the data distribution of the reference image, i.e., the high-resolution multispectral image X0, after noise addition to time t by formulas (1) and (2):

[0012]

[0013]

[0014] in α iis a predefined fixed parameter with a value range of (0,1); ∈ t is the noise information that obeys the standard Gaussian distribution; X0 is the reference image; X t represents the reference image with noise added to time t; I represents a matrix with all values ​​1; represents Gaussian distribution; q(X t |X0) represents the data distribution of the reference image X0 after adding noise to time t;

[0015] Step 3: Text modulation 3D convolutional neural network construction;

[0016] The input information received by the text modulation 3D convolutional neural network includes the time step t, the noisy image R t , text prompt L, full-color image P and multispectral image M, the prediction target of the network is a high-resolution multispectral image X0 without noise interference, that is, f θ (X t ,L,t,P,M)→X0;

[0017] The text modulation 3D convolutional neural network includes two branch networks, namely a conditional encoder branch and a U-type network branch. The conditional encoder branch extracts information of MS image blocks and PAN image blocks, and the U-type network branch extracts the noisy image X t The information is combined with the multi-scale features output by the conditional encoder to obtain the target high-resolution multispectral image; the conditional encoder branch and the U-type network branch both use the modulated 3D residual module as the basic module, which receives the encoding E at time step t t 、Code E of text prompt L l And the feature F of the previous layer j-1 , output feature F j ;

[0018] Step 3-1: Construct a time step encoding module;

[0019] The time step encoding module network depends on the time step t. The input time step t~{1,...,T} is converted into a one-dimensional vector using the sine-cosine encoding algorithm, and then a structure of sequential cascading of the Linear change layer and the LReLU activation function is used; the cascade structure cascades three sets of Linear change layers and LReLU, and outputs the processed one-dimensional time code E t ;

[0020] Step 3-2: Construct text encoding module;

[0021] The text encoding module consists of a pre-trained CLIP text encoder, a Linear linear change layer, and a cascade of LReLU activation functions. The module inputs a text prompt L and outputs a one-dimensional text encoding E. l ;

[0022] The text prompt L adopts a unified template, using the satellite name as the identifier of the satellite physical information. The specific template is "Images of <Class k >", where <Class k >Indicates the name of the satellite to be processed. For example, when the input panchromatic image P and multispectral image M are from QuickBird, the text prompt L is "Images of QuickBird";

[0023] The pre-trained CLIP text encoder uses the text encoder part of the CLIP-vit-large-patch14 model open sourced by Openai; after the text prompt L is input into the CLIP text encoder, a total of 77 dimensions 768 text encodings are obtained, of which <cls>After encoding, the input is a cascade structure of a Linear linear change layer and an LReLU activation function; the cascade structure includes three groups in total, and the final output is the processed one-dimensional text encoding E l ; The text encoding E t Input each modulated 3D residual module to modulate its output;

[0024] Step 3-3: Construct a modulated 3D residual module;

[0025] The modulated 3D residual module consists of a 3D convolution layer, a Linear linear change layer, an LReLU activation function and a skip connection; the input information includes the time code E t , text encoding E l With input feature F j-1 , output the processed feature F j ;

[0026] The 3D convolution layer uses a window size of 1×3×3, a padding of 1, and a step size of 1; the temporal code E t and text encoding E l The linear transformation layer changes the vector length to keep the same as the input feature F. j-1 The number of channels is the same, and the time coding E t With input feature F j-1 After addition, the structure is sequentially cascaded through LReLU, 3D convolution layer, LReLU, and 3D convolution layer; text encoding E l The weight of the second 3D convolutional layer is multiplied in the channel dimension to modulate the operation result of the module; the jump connection refers to the output result of the modulation 3D residual module and the input feature F j-1 After adding, output the final result F j ;

[0027] Step 3-4: Construct the conditional encoding network branch;

[0028] The conditional coding network branch is composed of a structure of three groups of modulated 3D residual modules, LReLU activation functions and downsampling layers cascaded; the conditional coding network branch receives the full-color image P and the multispectral image M as conditional information; specifically, the network input is P D -M r↑ , where P D Indicates that the panchromatic image P is copied to the same number of channels as the multispectral image M. r↑ It means that the multispectral image is upsampled r times to the same spatial resolution as the panchromatic image P, and the conditional input is obtained after the difference is calculated;

[0029] Condition Information D -M r↑ Before inputting the encoder, it needs to be reshaped into the (B, 1, C, H, W) format for the modulation 3D residual module to process; the downsampling layer uses a 3D convolutional layer with a window size of 1×3×3, a padding of 1, and a step size of 2; the conditional coding network outputs 3 sets of multi-scale features, and the feature shapes are (B, 32, C, H / 2, W / 2), (B, 64, C, H / 4, W / 4), and (B, 128, C, H / 8, W / 8);

[0030] Step 3-5: Construct U-shaped network branches;

[0031] The U-shaped network branch is composed of a modulated 3D residual module, an LReLU activation function, and a cascaded structure of a downsampling layer or an upsampling layer; the input of the U-shaped network branch is a reference image X that is noisy to time t t The three groups of multi-scale features extracted by the conditional coding branch are used to reconstruct the high-resolution multispectral image X0;

[0032] ImageX t Reshape to (B, 1, C, H, W) format before input for modulation 3D residual module processing; image X t Through the encoder structure of three groups of modulated 3D residual modules, LReLU activation function and downsampling layer cascade, and then through the decoder structure of three groups of modulated 3D residual modules, LReLU activation function and upsampling layer cascade, the decoder uses skip connection to concat encoder features and conditional encoder features of the same resolution, and finally outputs the reconstruction result through two modulated 3D residual modules;

[0033] Step 4: Training process;

[0034] The text modulation diffusion model is jointly trained on multiple remote sensing satellite datasets and uses the satellite-specific text prompts described in step 3-2; the training process uses K satellite datasets, denoted by D = D 1 ∪...∪D K , where D k The training data included is denoted as N k is the number of panchromatic-multispectral image pairs of the kth satellite dataset; the text modulation diffusion model sequentially samples batches of different satellite datasets for joint training in the joint dataset D, and the loss function used is as follows:

[0035]

[0036] Step 5: Reverse denoising process;

[0037] After the text modulation diffusion model training is completed, two samples X are sampled from the standard normal distribution based on the Markov chain process. T and z, use the following formula (4) to perform T-step iterations, and the iteration ends when t = 0. Finally, the Gaussian noise X T The fusion result of the panchromatic image P and the multispectral image M is obtained by denoising:

[0038]

[0039] in

[0040] Preferably, the total step length of the noise addition is T=1000.

[0041] Preferably, the number of satellite data sets K=3.

[0042] Preferably, the normalization process is to divide the pixel values ​​of all input images by 2047.0.

[0043] Preferably, the Wald protocol processing refers to first filtering the original MS and PAN images using a Gaussian smoothing kernel of size 5×5, and then downsampling to 1 / 4 of the original spatial resolution.

[0044] Preferably, the upsampled multispectral image M r↑ It means upsampling the MS image block to the spatial resolution of the PAN image block by bicubic interpolation, and the upsampling factor r is 4.

[0045] Preferably, the PAN image block size of the training set and validation set is 256×256, and the MS image block size is 64×64×4; the PAN image block size of the test set is 1024×1024, and the MS image block size is 256×256×4; the ratio of the data volume of the training set, validation set, and test set is 8:1:1.

[0046] The beneficial effects of the present invention are as follows:

[0047] The present invention provides a strongly generalized panchromatic sharpening method based on a text-modulated diffusion model, which can uniformly process satellite datasets with inconsistent numbers of spectral channels and train a unified model on a joint dataset. In the process of reverse denoising of the model, text prompts and observed images (including panchromatic images and multispectral images) are used as conditional information to guide the sampling of the diffusion model, thereby adapting to the differences between different satellite datasets and greatly enhancing the generalization ability of the model in unknown datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the method of the present invention.

[0049] Figure 2 This is the structural diagram of the text modulation 3D convolutional neural network of the present invention.

[0050] Figure 3 It is a comparison diagram between the present invention and other methods. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] The purpose of the present invention is to overcome the shortcomings of the above-mentioned existing technologies and propose a strong generalization panchromatic sharpening method based on a text modulation diffusion model to effectively utilize remote sensing satellite data sets with different physical properties and different numbers of spectral channels, and establish a universal model to improve the generalization ability.

[0053] A strong generalized panchromatic sharpening method based on a text modulation diffusion model comprises the following steps:

[0054] Step 1: Dataset preparation;

[0055] The data comes from QuickBird (QB), WorldView-4 (WV-4), and WorldView-2 (WV-2) satellite sensors. For QB data, the spatial resolution of PAN is 0.6 meters, the spatial resolution of MS is 2.4 meters, and MS contains 4 spectral bands: blue, green, red, and near-infrared bands. For WV-4 data, the spatial resolution of PAN is 0.3 meters, the spatial resolution of MS is 1.2 meters, and MS contains 4 spectral bands: blue, green, red, and near-infrared bands. For WV-2 data, the spatial resolution of PAN is 0.5 meters, the spatial resolution of MS is 2 meters, and MS contains 8 spectral bands: cyan, yellow, infrared, blue, green, red, near-infrared 1 band, and near-infrared 2 band. The spatial resolution ratio between MS and PAN images in the three datasets is 4.

[0056] The image block size of the training set and validation set is 256×256 (PAN) / 64×64×4 (MS), and the image block size of the test set is 1024×1024 (PAN) / 256×256×4 (MS). The ratio of training, validation, and test data is 8:1:1. Since there is no reference image, the MS image blocks and PAN image blocks in the training set, validation set, and test set are processed according to the Wald protocol, and then the processed images are used as the input of the network, and the original MS image is used as the reference image. The processed PAN image block is denoted as P, the MS image block is denoted as M, and the original MS image block is denoted as X.

[0057] The above three satellite datasets constitute a joint dataset for model training;

[0058] Step 2: Forward noise addition process;

[0059] The diffusion model noise adding process is as follows: Figure 1 As shown in Figure 1, the total step length of noise addition is set to T. For any time step t~{1,...,T}, the data distribution of the reference image X0 after noise addition to time t is directly calculated by formulas (1) and (2):

[0060]

[0061]

[0062] in α i is a predefined fixed parameter with a value range of (0,1); ∈ t is the noise information that obeys the standard Gaussian distribution; X0 is the reference image; X t represents the reference image with noise added to time t; I represents a matrix with all values ​​1; represents Gaussian distribution; q(X t |X0) represents the data distribution of the reference image X0 after adding noise to time t;

[0063] The α i Obey the linear noise strategy, that is, α i =1-β i , β i In [10 -6 ,10 -2 ] uniformly takes T values ​​within the range of i As i increases, α i It decreases as i increases;

[0064] Step 3: Text modulation 3D convolutional neural network construction;

[0065] like Figure 2 As shown, the input information received by the text modulation 3D convolutional neural network includes the time step t, the noisy image R t , text prompt L, full-color image P and multispectral image M, the prediction target of the network is the image X0 without noise interference, that is, f θ (X t ,L,t,P,M)→X0;

[0066] The text modulation 3D convolutional neural network comprises two branch networks, namely a conditional encoder branch and a U-type network branch. The conditional encoder branch extracts information of MS image blocks and PAN image blocks, and the U-type network branch extracts information of the noisy image X. t The information is combined with the multi-scale features output by the conditional encoder to obtain the target high-resolution multispectral image; the conditional encoder branch and the U-type network branch both use the modulated 3D residual module as the basic module, which receives the encoding E at time step t t , the code E of the text prompt L l And the feature F of the previous layer j-1 , output feature F j ;

[0067] Step 3-1: Construct a time step encoding module;

[0068] The text modulation 3D convolutional neural network depends on the time step t. The input time step t~{1,...,T} is converted into a one-dimensional vector using a sine-cosine coding algorithm, and then a linear change layer and an LReLU activation function are sequentially cascaded. The cascade structure cascades three sets of linear change layers and LReLU, and outputs the processed one-dimensional time code E t ;

[0069] Step 3-2: Build a text encoding module

[0070] The text encoding module consists of a pre-trained CLIP text encoder, a Linear linear change layer and a cascaded LReLU activation function. The module inputs a text prompt L and outputs a one-dimensional text encoding E. l ;

[0071] The text prompt L adopts a unified template, using the satellite name as the identifier of the satellite physical information, and the specific template is "Images of <Class k >", where <Class k >Indicates the name of the satellite to be processed. For example, when the input panchromatic image P and multispectral image M are from QuickBird, the text prompt L is "Images of QuickBird".

[0072] The pre-trained CLIP text encoder uses the text encoder part of the CLIP-vit-large-patch14 model open sourced by Openai; after the text prompt L is input into the CLIP text encoder, a total of 77 dimensions 768 text encodings are obtained, of which <cls>After encoding, the input is a cascade structure of a Linear linear change layer and an LReLU activation function; the cascade structure includes three groups in total, and the final output is the processed one-dimensional text encoding E l ; The text encoding E t Input each modulated 3D residual module to modulate its output;

[0073] Step 3-3: Construct a modulated 3D residual module;

[0074] like Figure 2 As shown, the modulated 3D residual module consists of a 3D convolution layer, a Linear linear change layer, an LReLU activation function and a skip connection; the input information includes the time code E t , text encoding E l With input feature F j-1 , output the processed feature F j .

[0075] The 3D convolution layer uses a window size of 1×3×3, a padding of 1, and a step size of 1; the temporal code E t and text encoding E l The linear transformation layer changes the vector length to keep the same as the input feature F. j-1 The number of channels is the same, and the time coding E t With input feature F j-1 After addition, the structure is sequentially cascaded through LReLU, 3D convolution layer, LReLU, and 3D convolution layer; text encoding E l The weight of the second 3D convolutional layer is multiplied in the channel dimension to modulate the operation result of the module; the jump connection refers to the output result of the modulation 3D residual module and the input feature F j-1 After adding, output the final result F j ;

[0076] Step 3-4: Construct the conditional encoding network branch;

[0077] like Figure 2 As shown in FIG. 1 , the conditional coding network branch is composed of a structure of three groups of modulated 3D residual modules, LReLU activation functions and downsampling layers cascaded; the conditional coding network branch receives the full-color image P and the multispectral image M as conditional information. Specifically, the network input is P D -M r↑ , where P D Indicates that the panchromatic image P is copied to the same number of channels as the multispectral image M. r↑ It means that the multispectral image is upsampled r times to the same spatial resolution as the panchromatic image P, and the conditional input is obtained after the difference is calculated;

[0078] The condition information P D -M r↑ Before entering the encoder, it needs to be reshaped into the (B, 1, C, H, W) format for modulation 3D residual module processing; the downsampling module uses a 3D convolutional layer with a window size of 1×3×3, a padding of 1, and a step size of 2; the conditional coding network outputs 3 groups of multi-scale features, and the feature shapes are (B, 32, C, H / 2, W / 2), (B, 64, C, H / 4, W / 4), and (B, 128, C, H / 8, W / 8).

[0079] Step 3-5: Construct U-shaped network branches;

[0080] like Figure 2 As shown, the U-shaped network branch consists of a modulated 3D residual module, an LReLU activation function, and a cascaded structure of a downsampling layer or an upsampling layer; the input of the U-shaped network branch is a noisy image X t The three groups of multi-scale features extracted by the conditional coding branch are used to reconstruct the high-resolution multispectral image X0;

[0081] Specifically, the noisy image X t Reshape to (B, 1, C, H, W) format before input for modulation 3D residual module processing; add noise image X t Through the encoder structure of three groups of modulated 3D residual modules, LReLU activation function and downsampling layer cascade, and then through the decoder structure of three groups of modulated 3D residual modules, LReLU activation function and upsampling layer cascade, the decoder uses skip connection to concat encoder features and conditional encoder features of the same resolution, and finally outputs the reconstruction result through two modulated 3D residual modules;

[0082] Step 4: Training process;

[0083] The text modulation diffusion model is jointly trained on multiple remote sensing satellite datasets and additionally uses the satellite-specific text prompts described in step 3-2; the training process uses K satellite datasets, denoted by D = D 1 ∪...∪D K , where D k The training data included is denoted as N k is the number of panchromatic-multispectral image pairs of the kth satellite dataset; the text modulation diffusion model sequentially samples batches of different satellite datasets for joint training in the joint dataset D, and the loss function used is as follows:

[0084]

[0085] Step 5: Reverse denoising process;

[0086] After the text modulation diffusion model training is completed, two samples X are sampled from the standard normal distribution based on the Markov chain process. T and z, use the following formula (4) to perform T-step iterations, and the iteration ends when t = 0. Finally, the Gaussian noise X T The fusion result of the panchromatic image P and the multispectral image M is obtained by denoising:

[0087]

[0088] in Specific embodiment:

[0090] (1) Dataset preparation:

[0091] Using panchromatic and multispectral images with a 4:1 aspect ratio and registered with each other, the following processing is performed:

[0092] ① Read the image and divide it into two parts, which are used as training data image and test data image respectively. The principle of division is to make the width of the two parts the same and the height ratio of 9:1. Both PAN and MS do this processing;

[0093] ② For the training data part, the image blocks of the corresponding positions of the matching PAN and MS training images are intercepted from left to right and from top to bottom, where the image block size of PAN is 256×256 and the image block size of MS is 64×64×4 (4 is the number of channels, which can be changed to 8 when the number of MS channels is 8). The test data part is constructed in a similar way, where the image block size of PAN is 1024×1024 and the image block size of MS is 256×256×4.

[0094] ③ Randomly divide 1 / 9 of the training data into validation set data.

[0095] So far, we have obtained the training set, validation set, and test set data. For QB, the training set contains 6943 pairs of images, the validation set contains 743 pairs of images, and the test set contains 156 pairs of images; for WV-4, the training set contains 7166 pairs of images, the validation set contains 772 pairs of images, and the test set contains 271 pairs of images; for WV-2, the training set contains 9641 pairs of images, the validation set contains 945 pairs of images, and the test set contains 136 pairs of images.

[0096] ④ When processing according to the Wald protocol, the PAN image and the MS image are Gaussian blurred with a Gaussian kernel of 5×5 and a standard deviation of 2, and then downsampled by 4 times as new images to form the training set. The same operation is performed on the validation set and the test set.

[0097] At this point, the dataset preparation step is completed.

[0098] (2) Network model construction

[0099] Network structure diagram Figure 2 , the important parameters for building the network include:

[0100] ① The 3D convolutional layer or 3D transposed convolutional layer used by the entire network: the window size of all convolutional layers is 1×3×3; the convolutional layer stride of the downsampling module is 2 and the padding is 1, the convolutional layer stride of the upsampling module is 2 and the padding is 1; the convolutional layer stride of other modules is 1 and the padding is 1.

[0101] Figure 2 The number of output channels of the middle convolutional layer is 32, 64, 128, 256, 128, 64, 32, 32, and 1 respectively; the skip connection occurs in the upsampling module and the fusion module, and the input information and the output of the downsampling module are transmitted to the second half of the network, connecting the features with the same resolution in the channel dimension;

[0102] ② The temporal coding layer uses the sine-cosine coding layer from the paper "Attention is all you need" to encode the time step t into a one-dimensional vector of length 32. This vector is converted into a vector of length 256 through the Linear layer. Before being added to the input of each module, the linear change layer is used to modify the number of channels to make the remaining input channels consistent.

[0103] (3) Network training

[0104] ① Input image: full-color image P, size 64×64 (height×width) and upsampled multispectral image M r↑ , size 64×64×4 (height×width×number of channels). Here the multispectral M is upsampled r↑ The multi-spectrum of size 16×16×4 is obtained by upsampling by 4 times through bilinear interpolation. The image input to the network is normalized, which means that the pixel value of the input image is divided by 2047.0.

[0105] ②Other related settings: Use Adam optimizer to update parameters. Set the number of training epochs to 200, the batch size to 32, and the initial learning rate to 0.0003. Use the validation set to test the network effect after every 50 epochs, and save the network parameters with the best effect. The joint dataset samples batch size data pairs from the training data of different datasets in sequence to train the model;

[0106] ③ Conditions for stopping training: the network’s loss function reaches convergence.

[0107] (4) Network testing

[0108] ① Input image: full-color image P, size 256×256 (height×width) and upsampled multispectral image M r↑ , size 256×256×4 (height×width×number of channels). Here the multispectral M is upsampled r↑ The multi-spectrum with size 64×64×4 is upsampled by 4 times through bilinear interpolation.

[0109] ② Load the best denoising network parameters saved during the training phase, select a pair of full-color image P and multispectral image M from the test set, generate text prompt L based on the data source of the test set, sample an image from Gaussian noise, and perform T-step repeated iterations based on the output of the modulated U-type network to obtain the fusion result.< / cls> < / cls>

Claims

1. A strong generalized panchromatic sharpening method based on a text modulation diffusion model, characterized in that: The steps include: Step 1: Dataset preparation; Image blocks are cut from left to right and from top to bottom for paired, registered remote sensing multispectral MS images and panchromatic images PAN, and these image blocks are divided into training set, validation set and test set; First, the training set, validation set, and test set are normalized; then the image blocks in the training set, validation set, and test set are processed according to the Wald protocol, and then the processed image blocks are used as the input of the model; The original MS image block is used as the reference image; Step 2: Forward noise addition process; Set the total step length of noise addition to T. For any time step t~{1, ..., T}, calculate the data distribution of the reference image, i.e., the high-resolution multispectral image X0, after noise addition to time t by formulas (1) and (2): in α i is a predefined fixed parameter with a value range of (0, 1); ∈ t is the noise information that obeys the standard Gaussian distribution; X0 is the reference image; X t represents the reference image with noise added to time t; I represents a matrix with all values ​​1; represents Gaussian distribution; q(X t |X0) represents the data distribution of the reference image X0 after adding noise to time t; Step 3: Text modulation 3D convolutional neural network construction; The input information received by the text modulation 3D convolutional neural network includes the time step t, the noisy image R t , text prompt L, full-color image P and multispectral image M, the prediction target of the network is a high-resolution multispectral image X0 without noise interference, that is, f θ (X t , L, t, P, M)→X0; The text modulation 3D convolutional neural network includes two branch networks, namely a conditional encoder branch and a U-type network branch. The conditional encoder branch extracts information of MS image blocks and PAN image blocks, and the U-type network branch extracts the noisy image X t The information is combined with the multi-scale features output by the conditional encoder to obtain the target high-resolution multispectral image; the conditional encoder branch and the U-type network branch both use the modulated 3D residual module as the basic module, which receives the encoding E at time step t t 、Code E of text prompt L l And the feature F of the previous layer j-1 , output feature F j ; Step 3-1: Construct a time step encoding module; The time step encoding module network depends on the time step t. The input time step t~{1,...,T} is converted into a one-dimensional vector using the sine-cosine encoding algorithm, and then a structure of sequential cascading of the Linear change layer and the LReLU activation function is used; the cascade structure cascades three sets of Linear change layers and LReLU, and outputs the processed one-dimensional time code E t ; Step 3-2: Construct text encoding module; The text encoding module consists of a pre-trained CLIP text encoder, a Linear linear change layer, and a cascade of LReLU activation functions. The module inputs a text prompt L and outputs a one-dimensional text encoding E. l ; The text prompt L adopts a unified template, using the satellite name as the identifier of the satellite physical information, and the specific template is "Images of <Class k >", where <Class k >Indicates the name of the satellite to be processed. For example, when the input panchromatic image P and multispectral image M are from QuickBird, the text prompt L is "Images of QuickBird"; The pre-trained CLIP text encoder uses the text encoder part of the CLIP-vit-large-patch14 model open sourced by Openai; after the text prompt L is input into the CLIP text encoder, a total of 77 dimensions 768 text encodings are obtained, of which <cls>After encoding, the input is a cascade structure of a Linear linear change layer and an LReLU activation function; the cascade structure includes three groups in total, and the final output is the processed one-dimensional text encoding E l ; The text encoding E l Input each modulated 3D residual module to modulate its output;< / cls> Step 3-3: Construct a modulated 3D residual module; The modulated 3D residual module consists of a 3D convolution layer, a Linear linear change layer, an LReLU activation function and a skip connection; the input information includes the time code E t , text encoding E l With input feature F j-1 , output the processed feature F j ; The 3D convolution layer uses a window size of 1×3×3, a padding of 1, and a step size of 1; the temporal code E t and text encoding E l The linear transformation layer changes the vector length to keep the same as the input feature F. j-1 The number of channels is the same, and the time code E t With input feature F j-1 After addition, the structure is sequentially cascaded through LReLU, 3D convolution layer, LReLU, and 3D convolution layer; text encoding E l The weight of the second 3D convolutional layer is multiplied in the channel dimension to modulate the operation result of the module; the jump connection refers to the output result of the modulation 3D residual module and the input feature F j-1 After adding, output the final result F j ; Step 3-4: Construct the conditional encoding network branch; The conditional coding network branch is composed of a structure of three groups of modulated 3D residual modules, LReLU activation functions and downsampling layers cascaded; the conditional coding network branch receives the full-color image P and the multispectral image M as conditional information; specifically, the network input is P D -M r↑ , where P D Indicates that the panchromatic image P is copied to the same number of channels as the multispectral image M. r↑ It means that the multispectral image is upsampled r times to the same spatial resolution as the panchromatic image P, and the conditional input is obtained after the difference is calculated; Condition Information D -M r↑ Before inputting the encoder, it needs to be reshaped into the (B, 1, C, H, W) format for the modulation 3D residual module to process; the downsampling layer uses a 3D convolutional layer with a window size of 1×3×3, a padding of 1, and a step size of 2; the conditional coding network outputs 3 groups of multi-scale features, and the feature shapes are (B, 32, C, H / 2, W / 2), (B, 64, C, H / 4, W / 4), and (B, 128, C, H / 8, W / 8); Step 3-5: Construct U-shaped network branches; The U-shaped network branch is composed of a modulated 3D residual module, an LReLU activation function, and a cascaded structure of a downsampling layer or an upsampling layer; the input of the U-shaped network branch is a reference image X that is noisy to time t t The three groups of multi-scale features extracted by the conditional coding branch are used to reconstruct the high-resolution multispectral image X0; ImageX t Reshape to (B, 1, C, H, W) format before input for modulation 3D residual module processing; image X t Through the encoder structure of three groups of modulated 3D residual modules, LReLU activation function and downsampling layer cascade, and then through the decoder structure of three groups of modulated 3D residual modules, LReLU activation function and upsampling layer cascade, the decoder uses skip connection to concat encoder features and conditional encoder features of the same resolution, and finally outputs the reconstruction result through two modulated 3D residual modules; Step 4: Training process; The text modulation diffusion model is jointly trained on multiple remote sensing satellite datasets and uses the satellite-specific text prompts described in step 3-2; the training process uses K satellite datasets, denoted by D = D 1 ∪...∪D K , where D k The training data included is denoted as N k is the number of panchromatic-multispectral image pairs of the kth satellite dataset; the text modulation diffusion model sequentially samples batches of different satellite datasets for joint training in the joint dataset D, and the loss function used is as follows: Step 5: Reverse denoising process; After the text modulation diffusion model training is completed, two samples X are sampled from the standard normal distribution based on the Markov chain process. T and z, use the following formula (4) to perform T-step iterations, and the iteration ends when t = 0. Finally, the Gaussian noise X T The fusion result of the panchromatic image P and the multispectral image M is obtained by denoising: in 2. The strong generalized pan-sharpening method based on the text modulation diffusion model according to claim 1, characterized in that: The total step length of the noise addition is T=1000.

3. The strong generalized pan-sharpening method based on text modulation diffusion model according to claim 1, characterized in that: The number of satellite data sets K=3.

4. The strong generalized pan-sharpening method based on text modulation diffusion model according to claim 1, characterized in that: The normalization process is to divide the pixel values ​​of all input images by 2047.

0.

5. The strong generalized pan-sharpening method based on text modulation diffusion model according to claim 1, characterized in that: The Wald protocol processing refers to first filtering the original MS and PAN images using a Gaussian smoothing kernel of size 5×5, and then downsampling them to 1 / 4 of the original spatial resolution.

6. The strong generalized pan-sharpening method based on text modulation diffusion model according to claim 1, characterized in that: The upsampled multispectral image M r↑ It means upsampling the MS image block to the spatial resolution of the PAN image block by bicubic interpolation, and the upsampling factor r is 4.

7. The strong generalized pan-sharpening method based on text modulation diffusion model according to claim 1, characterized in that: The PAN image block size of the training set and validation set is 256×256, and the MS image block size is 64×64×4; the PAN image block size of the test set is 1024×1024, and the MS image block size is 256×256×4; the ratio of the data volume of the training set, validation set, and test set is 8:1:1.

Images