Diffusion model for remote sensing panchromatic sharpening with convolution and controllable noise collaboration

Abstract

The application discloses a mixed convolution and controllable noise cooperative diffusion model remote sensing panchromatic sharpening method and belongs to the field of image processing. The application introduces a generative diffusion probability model into the field of multi-source remote sensing image fusion, improves a forward noise adding process of the diffusion model, and moves image residuals through a set displacement sequence and a noise scheduling strategy, so that the conversion process is effectively improved. Through a model of a layered optimized mixed convolution architecture, a fused image is gradually recovered from a noise image under the condition constraints of a panchromatic image and a low-resolution multispectral image. The encoder shallow layer adopts a depth separable convolution to efficiently extract spatial details, the deep layer adopts a dynamic shape convolution to improve multiscale feature representation capability, the decoder adopts a standard convolution to ensure reconstruction stability and spectral consistency, and cross-layer features are fused through a jump connection. The application significantly enhances spatial details of the diffusion model remote sensing panchromatic sharpening processing and effectively improves spectral fidelity.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a diffusion model remote sensing panchromatic sharpening method based on hybrid convolution and controllable noise cooperation. BACKGROUND

[0002] High-quality remote sensing images play a key role in many fields such as military and agriculture. However, the existing technology can only capture low-resolution multispectral (LRMS) images and high-resolution panchromatic (Pan) images. LRMS images retain rich spectral features, but their spatial resolution is limited. On the contrary, Pan images have higher spatial granularity, but they lack the separability of multispectral images due to being grayscale images. Panchromatic sharpening solves this contradiction through collaborative fusion to synthesize high-resolution multispectral (HRMS) images. Many panchromatic sharpening methods have been proposed, including traditional methods such as Component Substitution (CS), Multi-resolution Analysis (MRA), and Variational Optimization (VO), and deep learning (DL) methods including classic deep learning-based methods and diffusion model-based deep learning methods.

[0003] In the component substitution method, the low-resolution multispectral image is projected into a feature space based on spectral transformation, and the transformed components represent the original spatial information. Then, the blurred spatial information is replaced by the high-resolution Pan image, and finally, the inverse projection transformation is performed to obtain the fused image. The component substitution method can generate images with high spatial fidelity at a relatively low computational burden, but it causes severe spectral distortion in the fused image.

[0004] In the multi-resolution analysis method, the core idea is to perform multi-scale decomposition on the panchromatic image, obtain its high-resolution spatial information through a low-pass filter, and then inject it into the up-sampled low-resolution multispectral image to enhance its spatial details, thereby generating a high-resolution fused image. The multi-resolution analysis method has low computational complexity and excellent spectral fidelity, but the use of filtering leads to the loss of some spatial details, causing certain spatial distortion.

[0005] As a kind of remote sensing image fusion algorithm relying on mathematical modeling framework, the core of variational optimization method is to accurately construct the spatial and spectral information relationship model among low-resolution multispectral image, panchromatic image and unknown high-resolution multispectral image. This method introduces regularization variables to represent spatial and spectral information, establishes an energy function model based on regularization constraints, limits the spatial information of the fusion image to be consistent with the panchromatic image and the spectral information of the fusion image to be consistent with the multispectral image, and optimizes the model to obtain the fusion image. The variational optimization method has high computational complexity, but it can obtain good spatial and spectral fidelity.

[0006] The classical deep learning-based method usually takes the Pan image and the LRMS image as the fusion object to input the neural network. For example, a simple and effective three-layer convolutional network architecture, a network architecture composed of residual block stacking, and a neural network method using the difference between the information contained in the LRMS and Pan images, taking the difference between the up-sampled LRMS image and the Pan image as input. Although these methods effectively improve the performance, they generally rely on standard convolution operations for feature extraction and representation, and the standard convolution has two major defects: first, its sampling position is fixed in a square window of a certain size, which limits its deformation ability and makes it unable to adaptively find the sampling position; second, the number of sampling points of the convolution kernel is predetermined, making it difficult to adaptively capture features of different scales. In remote sensing images, the scale difference between different objects (such as small cars and large buildings) can be very significant, and standard convolution is not good at capturing these features, resulting in low efficiency of feature extraction.

[0007] In recent years, with the development of spatial adaptive convolution methods, some panchromatic sharpening methods using spatial adaptive convolution have also appeared, which can adaptively generate different convolution kernel parameters according to different spatial positions, making them adapt to different spatial regions. However, these improved convolution methods have not fully considered the rich scale information in remote sensing images.

[0008] At present, the generation model performs well on many computer vision tasks, mainly due to its powerful image reconstruction and generation capability, which is crucial for the recovery of detailed information in HRMS images. The deep learning method based on diffusion model takes the Pan image and the LRMS image as the fusion condition input into the neural network. The diffusion model (Denoising Diffusion Probabilistic Model, DDPM) is proposed as a discrete latent variable model for generating, including conditional or unconditional generation, text-to-image translation, image super-resolution, image restoration and other advanced image processing tasks. Among them, the diffusion model is parameterized by a Markov chain with a finite number of T time steps, and uses variational inference to gradually convert isotropic Gaussian noise into samples from the target distribution. DDPM contains two stages: a forward diffusion process and a backward denoising process. The forward diffusion process starts from the prior data distribution. Then, the Gaussian noise is continuously added to the prior data through the Markov chain process to obtain a noise image that approximates the standard normal distribution. The purpose of the reverse process of DDPM is to recreate samples in a specific data distribution by sampling randomly generated standard normal distribution Gaussian noise. This requires a reverse denoising process to learn the parameterized Gaussian transition and use the model to fit the mean and variance to approximate the conditional probability. Thanks to the rapid development of diffusion models, recently, diffusion models have also received more attention in the field of panchromatic sharpening.

[0009] At present, the deficiencies of the existing technology based on the diffusion model in dealing with the problem of panchromatic sharpening mainly include: 1) there are many sampling steps, which are slow, usually thousands of steps are needed to obtain a sample, which greatly limits the effective application of the model in actual scenarios. And most of them use standard convolution, whose sampling position is fixed in a square window of a certain size, which limits its deformation ability and makes it unable to adaptively find the sampling position. 2) The number of sampling points of the convolution kernel is predetermined, and it is difficult to adaptively capture features of different scales. In remote sensing images, the scale difference between different objects, such as small cars and large buildings, can be very significant, and standard convolution is not good at capturing these features, resulting in low feature extraction efficiency.

[0010] Therefore, in the process of dealing with the problem of panchromatic sharpening based on the existing technology based on the diffusion model, there is an urgent need for a diffusion model for remote sensing panchromatic sharpening that combines convolution and controllable noise. SUMMARY

[0011] In view of the deficiencies of the prior art, the application provides a diffusion model remote sensing panchromatic sharpening method based on mixed convolution and controllable noise cooperation, introduces a generative diffusion probability model into the field of multi-source remote sensing image fusion, improves a forward noise adding process of the diffusion model to greatly improve a conversion process, and enhances spatial details and spectral fidelity of a generated high-resolution multispectral image based on a constructed hierarchical optimization mixed convolution architecture (DS-UNet-SS).

[0012] The technical scheme adopted by the application is:

[0013] A diffusion model remote sensing panchromatic sharpening method based on mixed convolution and controllable noise cooperation comprises the following steps:

[0014] Step 1, input a panchromatic sharpening data set and perform data preprocessing on the panchromatic sharpening data set to obtain image data of each sample, including a high spatial resolution panchromatic image Pan, a low spatial resolution multispectral image ms and a reference image gt, wherein the reference image gt is a high spatial resolution multispectral image;

[0015] Step 2, perform forward degeneration of a conditional diffusion model based on a constructed Markov chain with a length of T steps, gradually degenerate the reference image gt to the ms image to obtain a noise image at each time step , wherein the time step , and The original multispectral image is the reference image gt.

[0016] Step 3, construct a fusion image prediction network deep model (i.e., a multiscale denoising deep model) and perform model parameter training; wherein the fusion image prediction network deep model is used to realize a reverse reconstruction process from the noise image to the original multispectral image .

[0017] The fusion image prediction network deep model comprises a double-branch encoder, a single-branch decoder, a feature fusion module and a global feature extraction module LEGM.

[0018] The double-branch encoder comprises a fusion branch and a noise branch, wherein the fusion branch takes the Pan image and the ms image as conditional inputs, and is used to extract spatial and spectral features of the image; the noise branch takes the noise image and the time step as inputs, and is used to extract time-embedded multispectral features; wherein the fusion branch comprises a plurality of cascaded first encoding units, the noise branch comprises a plurality of cascaded second encoding units, the number of the first encoding units is the same as that of the second encoding units; and the input of each second encoding unit further comprises the time step , which is subjected to a time condition modulation mechanism module to generate a spatially aligned modulation tensor.

[0019] The feature fusion module is configured to fuse the output features of the fusion branch and the noise branch to obtain first fusion features and input the single-branch decoder;

[0020] The input of the LEGM module includes a Pan image and an ms image to extract second fusion features by local feature embedding global feature and input the single-branch decoder;

[0021] The single-branch decoder is configured to decode and generate a high spatial resolution multispectral image, and includes a plurality of cascaded decoding units, the number of the decoding units being consistent with the number of the first / second encoding units; a skip connection is arranged between each decoding unit and the corresponding first encoding unit to receive the first encoding features of the same scale input by the first encoding unit;

[0022] When training the fusion image prediction network deep model, the input includes the Pan image and the ms image of the sample, and the noise image at any time and the corresponding noise image ; the model output is a predicted image of the reference image gt;

[0023] Step 4, generating a high spatial resolution multispectral image of a target image pair based on the trained fusion image prediction network deep model;

[0024] Wherein, the target image pair includes a Pan image and an ms image with the same image size;

[0025] The initial value of the time step t is set to T, which is decreased to 1 round by round, and the high spatial resolution multispectral image of each target image pair is generated after T rounds of iteration; wherein, the initial value of the noise image input into the noise branch is the ms image, and from the time step T-1, the input noise image is the high spatial resolution multispectral image output by the single-branch decoder in the last round.

[0026] Further, in step 1, the input panchromatic sharpening dataset includes a Pan image and a high spatial resolution multispectral image HRMS; the data preprocessing includes: cropping the Pan image and the HRMS image to obtain Pan images and HRMS images with consistent image sizes; then filtering and down-sampling the HRMS image to generate a low resolution multispectral image LRMS, and up-sampling the LRMS to obtain a low spatial resolution multispectral image ms consistent with the size of the Pan image; taking the Pan image and the ms image as the model input, and the cropped HRMS image as the reference image gt (i.e. the real label), thereby forming a panchromatic sharpening data pair for model training.

[0027] Further, in step 2, a hyperparameter and a displacement sequence is introduced to determine the noise schedule in the forward degradation process, which specifically includes:

[0028] the displacement sequence a noise coefficient of each time step , denotes a displacement of time step , and the noise coefficient of time step 1 is ; when t>1, ;

[0029] The boundary condition of the displacement of time step 1 is set based on hyperparameters , and the value of is selected;

[0030] The corresponding is calculated according to the formula , and the noise coefficient of the current time step is obtained according to ; ;

[0031] wherein the auxiliary parameter , the auxiliary parameter , wherein is a preset hyperparameter.

[0032] Further, the boundary condition of the displacement of time step 1 can be set as: .

[0033] Further, the fusion branch comprises, in sequence, a concatenation layer, a 3x3 convolutional layer, and first encoding units in cascade, and a down-sampling layer is arranged between adjacent first encoding units, for adjusting the size of first encoding features output by the first encoding units; wherein ;

[0034] The first encoding unit is a residual block in cascade of three, wherein each residual block comprises two first basic modules, and a residual connection is arranged between the output of the second first basic module and the input of the residual block; wherein the first basic module comprises, in sequence, a Swish activation function, a dropout layer Dropout and a depth separable convolutional layer.

[0035] Further, the noise branch comprises, in sequence, a 3x3 convolutional layer, and second encoding units in cascade, and a down-sampling layer is arranged between adjacent first encoding units, for adjusting the size of second encoding features output by the second encoding units; wherein ;

[0036] ​​​The second encoding unit is three cascaded first temporal embedding residual block modules, each of which includes two second basic modules and a temporal conditional modulation mechanism module; the input of the second second basic module is the sum fusion of the output of the first second basic module and the output of the temporal conditional modulation mechanism module, and a residual connection is arranged between the output of the second second basic module and the input of the first temporal embedding residual block module; wherein the second basic module sequentially includes: a Swish activation function, a dropout layer, and a depth separable convolution layer.

[0037] Further, the down-sampling layers of the first encoding unit and the second encoding unit adopt 2x2 max-pooling layers.

[0038] Further, the single-branch decoder sequentially includes: ) cascaded decoding units and a 1x1 convolution layer; wherein the input of each decoding unit includes a time step t;

[0039] The first decoding unit further includes a concatenation layer before the first decoding unit, which is used for feature concatenation of the output of the feature fusion module and the first encoding feature of the last first encoding unit of the fusion branch, and then the feature concatenation is taken as the input of the first decoding unit;

[0040] For the second decoding unit to the decoding unit, a sum layer, an up-sampling layer, and a concatenation layer are sequentially arranged between adjacent decoding units; wherein the sum layer is used for sum fusion of the second fusion feature output by the LEGM module and the decoding feature output by the previous decoding unit, to obtain a decoding fusion feature; the up-sampling layer is used for adjusting the size of the decoding fusion feature of the current stage; the concatenation layer is used for feature concatenation of the output of the up-sampling layer and the same-size first encoding feature of the first encoding feature output by the corresponding first encoding unit after the down-sampling layer, and then the feature concatenation is taken as the input of the next decoding unit;

[0041] The input of the 1x1 convolution layer is the sum fusion of the decoding feature output by the last decoding unit and the second fusion feature;

[0042] Each decoding unit is three cascaded second temporal embedding residual block modules, each of which includes two third basic modules and a temporal conditional modulation mechanism module; the input of the second third basic module is the sum fusion of the output of the first third basic module and the output of the temporal conditional modulation mechanism module, and a residual connection is arranged between the output of the second third basic module and the input of the second temporal embedding residual block module; the third basic module is: replacing the depth separable convolution layer in the first basic module with a convolution layer.

[0043] ​Further, the up-sampling layer of the decoding unit adopts the structure of bilinear / nearest neighbor interpolation and convolution layer to guarantee stable reconstruction of high-resolution spatial structure and spectral consistency, and inhibit fusion distortion.

[0044] Further, the feature fusion module sequentially comprises a splicing layer and two continuous dynamic shape convolution residual blocks, and the input of each continuous dynamic shape convolution residual block further comprises a time step The spatially aligned modulation tensor is generated through the time condition modulation mechanism module, wherein the continuous dynamic shape convolution residual block comprises two fourth basic modules and a time condition modulation mechanism module; the input of the second fourth basic module is the sum fusion of the output of the first fourth basic module and the output of the time condition modulation mechanism module, and a residual connection is arranged between the output of the second fourth basic module and the input of the continuous dynamic shape convolution residual block; the fourth basic module is: replacing the depth separable convolution layer in the first basic module with a dynamic shape convolution layer.

[0045] Further, the input of the LEGM module is set as the output of the splicing layer of the fusion branch, feature extraction is performed through a convolution layer and normalization processing is performed, and then the obtained features are respectively sent into two parallel branches to perform global feature extraction and local feature extraction; multi-head attention mechanism is adopted for global feature extraction, and depth separable convolution is adopted for local feature extraction; after sum fusion of the global feature extraction and the local feature extraction, the output of the linear mapping layer and the multilayer perceptron is sequentially passed through, and finally the output of the multilayer perceptron and the input of the LEGM module are sum fused through a residual connection to obtain the output of the LEGM module, so as to finally realize global feature modeling and efficient multi-source fusion under local perception guidance.

[0046] Further, the time condition modulation mechanism module sequentially comprises a time encoder and a multilayer perceptron, wherein the processing process of the time encoder comprises: according to the time step Corresponding noise coefficients are obtained ; the noise coefficients are represented as multi-dimensional vectors to generate a spatially aligned modulation tensor.

[0047] The technical scheme provided by the application at least brings the following beneficial effects:

[0048] In the training process of the fusion image prediction network deep model, a hierarchical optimization hybrid convolution architecture (DS-Unet-SS) is adopted, the down-sampling layer in the network adopts a depth separable convolution DSConv, so as to efficiently capture fine spatial details in a high-resolution input and has low calculation overhead; the middle bottleneck layer (i.e. the deep layer of the encoder) adopts a dynamic shape convolution DSSConv, which accurately adapts to multi-scale feature of ground objects through a dynamic generation mechanism of adaptive rectangular kernel and sampling points, so as to significantly improve the representation ability and adaptability of the model to the multi-scale and irregular geometric structure of ground objects in remote sensing images; the up-sampling layer (i.e. the decoder layer) adopts a standard two-dimensional convolution Conv2D, so as to guarantee stable reconstruction of high-resolution spatial structure and spectral consistency, and effectively suppress spatial distortion and spectral distortion commonly seen in the fusion process; cross-level feature fusion is realized through a skip connection, so as to complementarily fuse the features output by the shallow encoder rich in spatial details and the deep encoder rich in semantic context, and improve the integrity and accuracy of the decoder reconstruction.

[0049] A joint nonlinear scheduling strategy of noise intensity and displacement sequence is designed, which greatly improves the efficiency of the forward degradation process on the premise of ensuring that the low spatial resolution multispectral degradation result highly approximates the real observation distribution, thereby accelerating and stabilizing the reverse reconstruction process. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1 The flowchart of the image fusion method;

[0052] Figure 2 The forward noise adding and reverse denoising flowchart.

[0053] Figure 3 The structure diagram of the multiscale denoising deep model.

[0054] Figure 4 The structure diagram of the residual block (ResBlock1).

[0055] Figure 5 The structure diagram of the time embedding residual block 1 (TE-ResBlock1).

[0056] Figure 6 The structure diagram of the time embedding residual block 2 (TE-ResBlock2).

[0057] Figure 7A structural diagram of a dynamic shape convolution residual block (DSSConv-ResBlock).

[0058] Figure 8 A flowchart of a specific implementation of a dynamic shape convolution layer DSSConv. DETAILED DESCRIPTION

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will describe the technical solutions in the embodiments of the present application in a detailed and complete manner with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Generally, the components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed using different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application.

[0060] The embodiments of the present application provide a hybrid convolution and controllable noise collaborative diffusion model remote sensing panchromatic sharpening method, which is used for Pan image and LRMS image fusion to generate an HRMS image. The method of the present application comprises a structured forward degradation process and a conditional reverse reconstruction process. The forward degradation process is to construct a Markov chain to controllably and efficiently degrade a target HRMS image to an LRMS state. The key lies in designing a controllable noise scheduling plan, accurately regulating noise intensity and degradation step length through a dynamic nonlinear scheduling function, significantly optimizing the efficiency and fidelity of the degradation trajectory, and laying a foundation for subsequent high-quality reconstruction. The reverse reconstruction process is to generate a conditional image by using a double-flow encoder-single-flow decoder Unet architecture (referred to as DS-UNet-SS) to reconstruct a high-fidelity HRMS image.

[0061] In one embodiment, as shown in Figure 1 The hybrid convolution and controllable noise collaborative diffusion model remote sensing panchromatic sharpening method provided by the embodiments of the present application comprises:

[0062] Step S1, data preprocessing, performing data preprocessing on the input panchromatic sharpening data set, and then dividing it into a training set and a test set according to the set allocation ratio; wherein the image of each sample after data preprocessing includes: a high spatial resolution panchromatic image Pan, a low spatial resolution multispectral image ms, and a reference image gt (a high spatial resolution multispectral image, i.e., a true label).

[0063] Step S2, performing a forward degradation process of a conditional diffusion model based on the training set, controllably and efficiently degrading a target high spatial resolution multispectral image to a low spatial resolution multispectral image based on a constructed Markov chain with a length of T steps;

[0064] Step S3, a fusion image prediction network depth model (i.e., a multi-scale denoising depth model) is constructed and model parameter training is performed; wherein the fusion image prediction network depth model is used to realize the reverse reconstruction process from the noise image to the original multi-spectral image (i.e., the reference image gt).

[0065] When training the multi-scale denoising depth model, the input includes the Pan image and the ms image of the sample, as well as the noise image (T~1) and the corresponding noise image at any time step; the model output is the predicted image , so that the model parameters are trained according to the prediction loss between the predicted image and the reference image gt. When the training termination condition set by humans (such as the upper limit of the number of training times or the convergence of the loss function value) is met, the training stops, and the optimal model is obtained. Further, the model effect can be evaluated based on the test set.

[0066] Step S4, based on the trained fusion image prediction network depth model, high spatial resolution multi-spectral images of the target image pair (Pan image and ms image) are generated; at this time, the initial value of the time step t is set to T, the initial value of the noise image input into the noise branch is the ms image, which is decreased to 1 round by round, and the high spatial resolution multi-spectral images of each target image pair are generated after T rounds of iteration. For the current time step t, the high spatial resolution multi-spectral image output by the model can be defined as When t-1 is not 0, then is updated first, and then is assigned to , and then the next round of forward inference process is started based on the , the time step t, the Pan image and the ms image; if t-1 is 0, the high spatial resolution multi-spectral image of the target image pair is obtained based on the output . Thus, the reverse reconstruction is realized based on the network model (i.e., the multi-scale denoising depth model) constructed by the DS-UNet-SS architecture of the embodiment of the present application.

[0067] The network model adopts a Unet architecture of a double-flow encoder-single-flow decoder, the double-flow encoder includes two parallel branches: a fusion branch and a noise branch (a denoising branch); the fusion branch takes the Pan image and the LRMS as conditional input, and extracts spatial-spectral features; the denoising branch inputs the noise image and the time embedding, and focuses on recovering the target multi-spectral features from the degradation state; the features extracted by the two branches are spliced at the end of the encoder, and are input into the output of a local feature embedded global feature extraction module (LEGM) together, for upsampling and feature integration in the single-flow decoder, to reconstruct a high-fidelity HRMS image.

[0068] In one embodiment, the data preprocessing of step S1 specifically includes:

[0069] Read the panchromatic sharpening training set and randomly divide it into B batches, each batch containing b data, each data containing a high spatial resolution panchromatic image Pan, a reference image gt (a high spatial resolution multi-spectral image), and a low spatial resolution multi-spectral image LRMS, wherein the image dimensions are: Pan∈ , gt∈ , LRMS∈ ; upsample the LRMS four times to obtain a low spatial resolution multi-spectral image , wherein H, W, and C represent the length, width, and channel number of the image, respectively, , and the real number field is represented by R;

[0070] In this embodiment, the model is trained on a WorldView-2 (eight-band) satellite data set. 46635 pairs of Pan / LRMS / gt images with sizes of 128×128×1, 32×32×8, and 128×128×8 are simulated respectively, and then they are used as training set data. In this embodiment, since the ideal high-resolution multi-spectral fusion result (gt) is unknown, it is obtained through the following processing:

[0071] First, the original large-size image is regularly cropped to obtain a high-resolution panchromatic image pan and a high-resolution multi-spectral image HRMS; then, the HRMS image is degraded according to the Wald protocol, a low-resolution multi-spectral image LRMS (with dimensions of ) is generated by bicubic filtering and downsampling with a resolution factor of 4, and a multi-spectral image ms with the same size as the Pan image is obtained by using a polynomial kernel with 23 coefficients to perform 4 times upsampling on the generated LRMS image; finally, the pan image and the ms image are used as model input, and the original HRMS image is used as the true label gt, to form a panchromatic sharpening data pair for training.

[0072] In one embodiment, the forward degradation process of the conditional diffusion model in step S2 specifically comprises:

[0073] First, define to represent a high spatial resolution multispectral image, i.e. a reference image , to represent an input low-resolution multispectral image ms, define the image residual as , construct a Markov chain with T steps to gradually move their residuals , so that transition to . As shown in Figure 2 , a forward step is defined as follows:

[0074] (1)

[0075] (2)

[0076] (3)

[0077] (4)

[0078] wherein represents a multispectral image with noise added to the t-th moment, , is an identity matrix, represents the transition probability distribution of the forward degradation process, is a normal distribution with mean and variance . Introduce hyperparameters and displacement sequence to determine the noise scheduling in the diffusion process.

[0079] The specific construction process is as follows: based on the hyperparameter , the noise intensity of the entire transition process is regulated. When is very small, the final state converges to a perturbation around the image, shortening the Markov chain.

[0080] The displacement sequence contains the noise coefficient of each time step, which monotonically increases with the time step t, , , is the noise coefficient at the t-th moment, and when t>1, .

[0081] Because formula (4) shows that the noise level of state and is proportional to the boundary condition is set to , ensure ; the termination step is 0.999, so that it meets ; set the intermediate step geometry scheduling as shown in formula (5) - (7), the hyperparameter controls the growth rate of .

[0082] (5)

[0083] Where: the parameters , are specifically set as:

[0084] (6)

[0085] (7)

[0086] In an embodiment, the network structure of the multiscale denoising deep model adopted by the embodiment of the present application is shown in Figure 3 , which includes a double-branch encoder, a single-branch decoder, a feature fusion module and a LEGM module; wherein the double-branch encoder includes a fusion branch and a noise branch, the fusion branch takes the prior panchromatic image Pan and low-resolution multispectral image ms as conditional input, aiming to fully extract the rich spatial and spectral features in the image; the noise branch uses noise image and diffusion step time embedding, striving to obtain multispectral features; the features obtained by the two branches are spliced to form a feature representation of an approximate fused image. In the decoder part, the input includes the spliced output from the double-branch encoder, as well as the output of the local feature embedding global feature extraction module LEGM, after decoder processing, a multispectral fused image with high spatial resolution is finally generated, that is, the predicted initial image .

[0087] As Figure 3As shown, the structure of the fusion branch includes, in sequence, a concatenation layer, a 3x3 convolution layer, and a plurality of cascaded first encoders; wherein each first encoder is composed of 3 cascaded residual blocks (ResBlock1), and a down-sampling layer is arranged between adjacent first encoders, for adjusting the size of the output first encoded features; in this embodiment, the number of first encoders of the fusion branch is set to 5. The noise branch includes, in sequence, a 3x3 convolution layer and a plurality of cascaded second encoders; wherein each second encoder is composed of 3 cascaded time embedding residual blocks 1 (TE-ResBlock1), and a down-sampling layer is arranged between adjacent second encoders, for adjusting the size of the output second encoded features. In this embodiment, the number of second encoders is set to 5. And the input of each second encoder of the noise branch further includes a spatially aligned modulation tensor generated by the time embedding t through a time-conditioned modulation mechanism module (TimeEmbedding+MLP). That is, the time-conditioned modulation mechanism module is composed of a time encoder and a multi-layer perceptron.

[0088] The feature fusion module includes, in sequence, a concatenation layer and two consecutive dynamic shape convolution residual blocks DSS-ResBlock; and the input of each consecutive dynamic shape convolution residual block further includes a spatially aligned modulation tensor generated by the time embedding t through a time-conditioned modulation mechanism module.

[0089] The single-branch decoder includes, in sequence, 5 cascaded decoding units and a 1x1 convolution layer; wherein the decoding unit is composed of 3 cascaded time embedding residual blocks 2 (TE-ResBlock2), and the input of each decoding unit includes the time step t;

[0090] The first decoding unit further includes a concatenation layer, for performing feature concatenation on the output of the feature fusion module and the first encoded features of the fifth first encoding unit of the fusion branch, and then taking the result as the input of the first decoding unit;

[0091] For the 2nd to 5th decoding units, a summing layer, an up-sampling layer, and a concatenation layer are sequentially arranged between adjacent decoding units; wherein the summing layer is used for summing and fusing the second fusion features output by the LEGM module and the decoding features and the second fusion features output by the previous decoding unit, to obtain the decoding fusion features of the current stage; the up-sampling layer is used for adjusting the size of the decoding fusion features of the current stage; and the concatenation layer is used for performing feature concatenation on the output of the up-sampling layer and the same-size first encoded features of the corresponding first encoding unit after the down-sampling layer, and then taking the result as the input of the next decoding unit; that is, the same-size encoded features input through the skip connection are fused to compensate for the information loss; the input of the 1x1 convolution layer is the summing and fusion of the decoding features and the second fusion features output by the last decoding unit.

[0092] The LEGM module takes the spliced result (spliced in the channel dimension) of the prior full-color image Pan and the low-resolution multispectral image ms as input, first extracts local features through standard two-dimensional convolution and performs normalization processing; then, the features are sent into two parallel branches: one branch uses a multi-head attention mechanism to capture global dependencies, and the other branch uses a depth separable convolution to further extract local context information; the output results of the two branches are added and fused, and then pass through a linear mapping layer and a multilayer perception to further enhance the feature expression capability; the entire module embeds a residual connection structure as a jump branch, effectively promoting feature reuse, that is, the input of the LEGM module and the output of the multilayer perception are added and fused to obtain the output of the LEGM module, so as to finally realize global feature modeling and efficient multi-source fusion under the guidance of local perception.

[0093] In this embodiment, the structure of the residual block (ResBlock1) is as shown in Figure 4 , and specifically includes two first basic modules (Block 1,1 and Block 1,2 ), and a residual connection is arranged between the output of Block 1,2 and the input of the residual block; wherein Block 1,1 and Block 1,2 modules include in turn: a Swish activation function, an optional regularization Dropout layer, and a depth separable convolution DSConv. In this residual block design, the residual connection uses a conditionally adaptive residual path, which uses an identity mapping to maintain direct connection when the input and output channel numbers are the same, and automatically enables a 1x1 convolution for feature projection when the dimensions do not match to ensure mathematical additivity.

[0094] In this embodiment, the time embedding residual block 1 (TE-ResBlock1) is created on the basis of the ResBlock1 module, and a time encoder and a multilayer perceptron MLP are used to process the diffusion step time embedding t. As shown in Figure 5 , the TE-ResBlock1 module includes two first basic modules (Block 1,1 and Block 1,2 ) and a time condition modulation mechanism module; wherein the input of Block 1,2 is the addition and fusion of the output of the first Block 1,1 module and the output of the time condition modulation mechanism module, and a residual connection is arranged between the output of Block 1,2 and the input of the TE-ResBlock1 module.

[0095] In this embodiment, the structure of the time embedding residual block 2 (TE-ResBlock2) is as shown in Figure 6As shown, including two second basic modules (Block 2,1 and Block 2,2 ) and a time condition modulation mechanism module; wherein, the input of Block 2,2 module is the sum fusion of the output of Block 2,1 module and the output of the time condition modulation mechanism module, and the residual connection is set between the output of Block 2,2 module and the input of TE-ResBlock2 module. Wherein, Block2 module includes in turn: Swish activation function, an optional Dropout layer, a standard two-dimensional convolution layer Conv2D. That is, TE-ResBlock2 module is to replace Block1 module in TE-ResBlock1 module with Block2 module.

[0096] In this embodiment, the dynamic shape convolution residual block DSSConv-ResBlock is then based on TE-ResBlock1, replacing the depth separable convolution layer therein with a dynamic shape convolution layer, as shown in Figure 7 As shown, including two third basic modules (Block 3,1 and Block 3,2 ) and a time condition modulation mechanism module; wherein, the input of Block 3,2 module is the sum fusion of the output of Block 3,1 module and the output of the time condition modulation mechanism module, and the residual connection is set between the output of Block 3,2 module and the input of DSSConv-ResBlock module. Wherein, Block3 module includes in turn: Swish activation function, an optional Dropout layer, a dynamic shape convolution layer.

[0097] In this embodiment, the processing of time embedding t is specifically:

[0098] Inspired by the Transformer position encoding, the DS-UNet-SS network introduces a special time encoder, which first obtains the corresponding noise scheduling parameter according to the input time embedding t, and then converts into a multi-dimensional vector representation TE, and the specific transformation formula is:

[0099] (8)

[0100] Wherein, TE (Time Embedding) represents the final generated time embedding vector, is the connected product of all parameters from step 0 to step t, that is , which represents the total noise progress from the starting point to the current time point. 2i and 2i+1 are index parameters of the function, which are used to specify which dimension in the time embedding vector TE is to be calculated. The value range of is limited to [0, ], represents the preset dimension of the vector. The multi-dimensional time embedding vector TE obtained through the above transformation is further input into a multi-layer perceptron (MLP) composed of a linear layer and a Swish activation function for processing. The MLP maps the time vector to a dimension matching the number of network feature channels, and then expands it to the same spatial size as the convolution feature map through a broadcast mechanism, and finally element-wise adds the feature map output by the first convolution block of the time embedding residual block. This mechanism ensures that the time step information is effectively embedded into the feature extraction process, enabling the model to adaptively adjust the feature response according to different denoising stages, thereby supporting dynamic and diffusion progress-adaptive feature fusion and noise prediction.

[0101] In the method of the embodiment of the present application, the generative diffusion probability model is introduced into the field of multi-source remote sensing image fusion, the forward noise adding process of the diffusion model is improved, the image residual is moved through the set displacement sequence and noise scheduling strategy, so that the model completes the noise adding process within a limited number of steps, greatly improving the conversion process. Through training of the DS-Unet-SS network (i.e. a multi-scale denoising deep model), maximum likelihood estimation is performed, and under the condition constraints of panchromatic images and low-resolution multispectral images, the fused image is gradually recovered from the noise image. The DS-Unet-SS network adopts a hierarchical optimization hybrid convolution architecture. The shallow encoder layer uses a depth separable convolution to efficiently capture fine spatial details in high-resolution inputs and reduce computational overhead. The deep encoder layer uses a dynamic shape convolution, which significantly improves the model's ability to represent and adapt to the geometric structure of multi-scale and irregular features in remote sensing images by adaptively generating a rectangular convolution kernel and dynamic sampling points. The decoder layer uses a standard two-dimensional convolution to ensure stable reconstruction of high-resolution spatial structures and strict spectral consistency, effectively suppressing common spatial distortion and spectral distortion during the fusion process. Cross-layer feature fusion is achieved through a skip connection, which complements the features output by the shallow encoder rich in spatial details and the deep encoder rich in semantic context, improving the integrity and accuracy of the decoder reconstruction. Through the above improvements, the diffusion probability model is better adapted to the remote sensing image fusion task, and the performance of the model in fusing remote sensing images is improved.

[0102] In one embodiment, in step S3, the model training specifically includes:

[0103] Step 3.1 initializes the number of training times.

[0104] Step 3.2 Determine the model input Pan, ms, t and ;

[0105] Step 3.3 Perform fusion image prediction network depth model training: the network adopts a dual-branch encoder and a single-branch decoder architecture, and the dual-flow encoder is divided into a noise branch and a fusion branch. The network first uses a 3x3 convolutional layer to extract shallow features of the noise image and the prior image Pan+ms.

[0106] The fusion branch processes the splicing features of the panchromatic image Pan and the up-sampled multispectral image ms, and uses a 5-level residual block (ResBlock1) to extract spatial-spectral joint features.

[0107] The noise branch receives the shallow features of the noise image and the diffusion step time embedding t as input, and extracts noise-related features through a 5-level time embedding residual block 1 (TE-ResBlock1). The time embedding t is generated by a spatial alignment modulation tensor through a time condition modulation mechanism module, and is injected into the noise feature space through feature element-wise addition, replacing the traditional AdaGN. The TE-ResBlock1 module uses a depth separable convolution DSConv to optimize efficiency. Both branches are progressively down-sampled by 2x2 max pooling.

[0108] The dual-branch features are spliced at the bottleneck layer and then subjected to multi-source feature depth fusion through two consecutive dynamic shape convolution residual blocks DSS-ResBlock. The DSS-ResBlock replaces the depth separable convolution layer in the TE-ResBlock1 with a dynamic shape convolution layer.

[0109] The decoder input includes the spliced output from the dual-branch encoder and the output of the local feature embedding global feature extraction module LEGM. The decoder uses a symmetric 5-level time embedding residual block 2 (TE-ResBlock2) structure, in which the TE-ResBlock2 replaces the depth separable convolution layer in the TE-ResBlock1 with a Conv2D convolution layer. Then, through bilinear interpolation followed by a Conv2D layer, 2x up-sampling is achieved, and the same scale encoder features are combined through a skip connection to compensate for information loss. Finally, a 1x1 convolution block is used to map the number of channels to the target number of bands, obtaining the fused image.

[0110] The L1 loss function is used to control the model training, and after obtaining the loss function, the Adam optimizer is used to update the model parameters.

[0111] The dynamic shape convolution layer DSSConv precisely adapts to multi-scale feature objects through an adaptive rectangular kernel and a dynamic sampling point generation mechanism, as shown inFigure 8 The specific implementation process is as follows:

[0112] (1) Dynamic learning of convolution kernel height and width

[0113] The learning process is expressed by mathematical formula as:

[0114] (9)

[0115] wherein, the input feature map wherein, represents the number of channels of the input feature map, corresponding to two sub-networks responsible for predicting the height and width of the convolution kernel, and each sub-network structure is composed of a shared feature extractor and an independent height / width learner, represent relevant parameters. The output feature map is expressed as , is a height feature map, is a width feature map.

[0116] The last layer of the height and width learner is a Sigmoid function, wherein , therefore , representing the relative size of the convolution kernel.

[0117] The value range thereof is constrained in the following manner:

[0118] (10)

[0119] wherein, and are modulation factors for constraining the range of the feature map height and width, limiting the kernel height to and the width to .

[0120] (2) Adaptive selection of the number of sampling points

[0121] In order to ensure that the convolution kernel has a clear center position, the number of sampling points needs to be an odd number, and the input height and width feature maps and , first calculate the global average value: , .

[0122] Then determine the number of sampling points as: , wherein, represents rounding down, if the result is an odd number, it remains unchanged, if the result is an even number, it is reduced by 1. Among them, the function is used to return a value not greater than its function input the maximum odd number to ensure the number of sampling points is odd, optimizing the symmetry and coverage of sampling. m and n represent the modulation coefficients mapping the height and width of the convolution kernel to the number of sampling points, given the fixed height and width of the convolution kernel, the larger the values of m and n, the fewer the sampling points, and the sparser their distribution. The total number of sampling points is . The final output dynamic sampling point number .

[0123] (3) Generating a sampling map

[0124] Input: feature map ;

[0125] Spatial position traversal and standard grid generation: traverse each spatial position of the input feature map , where the value range of the horizontal and vertical coordinates is: , . Take

[0126] as the coordinate origin, and generate a standard offset grid according to the preset number of sampling points . The element in the i-th row and j-th column of the grid is defined as:

[0127] (11)

[0128] Adaptive position parameter prediction and scaling matrix construction: predict the exclusive parameters of the current position through sub-networks : convolution kernel height and width , generate a scaling matrix , where the element in the i-th row and j-th column of is:

[0129] (12)

[0130] Adaptive offset matrix calculation: element-wise multiplication of the scaling matrix and the standard grid G to generate an adaptive offset matrix centered at :

[0131] (13)

[0132] where the element in the i-th row and j-th column of is:

[0133] ​​​​(14)

[0134] Bilinear interpolation and sampling map generation: for each element of the adaptive offset matrix , compute its absolute sampling coordinates on the input feature map : , where , , is usually a non-integer coordinate used to represent the target point.

[0135] Get the four nearest integer grid point coordinates around it, two points in the horizontal direction: , ; two points in the vertical direction: , ; where .

[0136] Compute the normalized weights:

[0137] (15)

[0138] where and represent the normalized weights in the horizontal and vertical directions, respectively, used to represent the offset ratio of the target point in the horizontal and vertical directions relative to the top-left corner of the grid cell it is in , . Since = 1, = 1, and are just decimals in the range [0, 1), which determine the contribution of the four pixels around the target point, so that the interpolation result can smoothly reflect the characteristics at non-integer coordinates.

[0139] Extract the pixel values of the four neighboring points from the input feature map :

[0140] , , , ,

[0141] According to the interpolation formula, we get:

[0142] (16)

[0143] Construct the sampling map tensor: define the sampling grid corresponding to each position as:

[0144] (17)

[0145] Integrate global sampling map: stitch all grid along spatial dimension to get spatially warped sampling map :

[0146] (18)

[0147] where, is the sampling grid at position .

[0148] (4) Adaptive convolution and affine transformation

[0149] To introduce spatial adaptivity, affine transformation is applied to the output feature map. The dimensions of input sampling map , convolution kernel weight , are dynamically determined by , avoiding the limitation of fixed kernel size. Among them denotes the number of output channels set; the affine transformation matrix and bias are predicted by the lightweight network , and the convolution operation is performed on the sampling map , and the formula is defined as:

[0150] (19)

[0151] The spatial adaptive affine transformation is applied to the convolution output :

[0152] (20)

[0153] Thus, the final feature map is obtained.

[0154] In one embodiment, step S4 is to input Pan, ms, and and t into the fusion image prediction network depth model obtained in step 3, and learn the distribution of using the depth model, as follows:

[0155] (21)

[0156] where, and are the mean and variance of distribution, respectively, is the intrinsic parameter of the model; the mean​​ and variance The calculation formula is:

[0157] (22)

[0158] (23)

[0159] wherein, is a deep neural network with parameters, and the purpose of introducing this network is to use it to predict , and the objective function is , wherein, . . is an adaptive weight that changes with time step t, and the prediction error at different time steps has different contributions in the total loss function.

[0160] Given a low-resolution multispectral image , obtain from the conditional distribution formula (3), and then use the deep model to learn the parameterized inverse transition kernel , after T-step iteration, generate the output from .

[0161] The present application evaluates the effectiveness of the method on 8-band data obtained by WorldView2 (WV2 for short) sensor, sets up a hyperparameter setting experiment, and studies the hyperparameters including the diffusion step number T required by the Markov chain, the hyperparameter p capable of flexibly controlling the residual transition speed, and the hyperparameter K capable of controlling the noise intensity in the state . The experimental results are shown in Table 1, and the final selection of the hyperparameters T=15, p=0.3, and K=0.0001 as the optimal configuration.

[0162] The following optimized training configuration is adopted in the embodiment: the AdamW optimizer is used to iteratively optimize the neural network parameters, sufficient training is performed for 150 epochs, the learning rate is set to 0.0001; the L1 norm loss (Smooth L1 Loss) is selected as the loss function, which makes the model stably converge to the optimal solution in the later training period.

[0163] In order to obtain more comprehensive and objective analysis conclusion, the multi-angle evaluation strategy is adopted in the embodiment of the present application: on the data level, the two types of evaluation indexes of the reference image and the non-reference image are comprehensively used to evaluate the fusion result from different dimensions; on the perception level, the subjective evaluation of the fusion result is directly carried out by means of the human eye, and meanwhile the difference between the fusion image and the reference image is directly highlighted by using the difference image, and the multi-dimensional evaluation framework is helpful to more objectively and accurately measure the actual performance of different fusion algorithms. The quantitative evaluation indexes used in the present application are defined as follows.

[0164] The reference image evaluation indexes used in the present application include: peak signal-to-noise ratio (PSNR), universal image quality index (UIQI), spectral angle mapper (SAM), global relative spectral loss (ERGAS) and spatial correlation coefficient (SCC). The peak signal-to-noise ratio (PSNR) is a comprehensive evaluation index for evaluating the image reconstruction quality. The greater the PSNR value is, the better the overall quality of the image is. The universal image quality index (UIQI) is a comprehensive index for globally evaluating the fusion image, and the value range is between 0 and 1. The greater the index tends to 1, the greater the similarity between the two images is, and the better the effect of the fusion image is. The spectral angle mapper (SAM) is an important index for evaluating the spectral consistency between the fusion image and the reference image. The ideal value is 0, which indicates that the spectral information of the two images is completely consistent. The global relative spectral loss (ERGAS) is a comprehensive index for globally evaluating the fusion image, and the smaller the value is, the higher the similarity between the fusion image and the reference multispectral image is, and the better the fusion effect is. The spatial correlation coefficient (SCC) is an index for evaluating the spatial structure similarity between the fusion image and the reference image, and the statistical correlation of the local spatial features between the images is calculated to quantify the consistency. The ideal value of the index is 1. The test results of the embodiment are shown in Table 1, and the best results are obtained in each objective index.

[0165] Table 1: Index comparison table of different models

[0166]

[0167] Wherein, SFIINet is a panchromatic sharpening network based on space-frequency domain information integration, PanNet is a panchromatic sharpening network, FusionNet is a panchromatic sharpening fusion network, LACNet is a panchromatic sharpening network based on local context adaptive convolution kernel of global harmonic bias, GPPNN is a panchromatic sharpening network based on gradient projection, CIKANet is a panchromatic sharpening network based on cross interaction of convolution kernel attention, FGF-GAN is a panchromatic sharpening network based on light generation adversarial network of fast guided filtering, MSDDN is a panchromatic sharpening network based on multi-scale dual-domain guided network, Diffusion is a diffusion probability model, and ResDualDiff is a panchromatic sharpening fusion method based on residual shift of double-branch conditional diffusion model.

[0168] Table 1 shows the comparison results of the indicators of the data set of the present application with the existing 10 image fusion methods on WorldView2, and from Table 1, it can be seen that the present application has obtained the optimal value in all indicators.

[0169] The objective evaluation indicators of the present application for the no-reference image include spectral distortion index (Spectral Distortion Index), spatial distortion index (Spatial Distortion Index) and no-reference image quality index (Quality with No Reference, QNR). The spectral distortion index is an index for measuring the spectral distortion degree of the fused image, and is based on the difference of the correlation between the bands. The smaller the value of the index is, the better the spectral fidelity of the fused image is. The spatial distortion index is an index for measuring the spatial structure distortion degree of the fused image, and is calculated by comparing the local variance. The smaller the value of the index is, the better the spatial fidelity of the fused image is. The QNR is an index for comprehensively evaluating the image fusion effect, and the smaller the spectral distortion and spatial structure distortion degree of the fused image is, the closer the index is to the ideal value 1.

[0170] The no-reference image quantitative indicators of the present application and other 10 algorithms on WorldView2 data set are as shown in Table 2.

[0171] Table 2 is a comparison table of no-reference image quantitative indicators of each model on WorldView2 data set.

[0172]

[0173] ​​Based on the index results shown in Table 2, overall, CIKANet and ResDualDiff and the model of the application can better retain the spatial detail information of the panchromatic image, and the application achieves the best fusion effect by comprehensively considering the spatial detail and spectral information, thereby further verifying the effectiveness of the algorithm.

[0174] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0175] The above only describes some embodiments of the present application. For those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A diffusion model remote sensing panchromatic sharpening method combined with mixed convolution and controllable noise, characterized in that, The method comprises the following steps: Step 1, inputting a panchromatic sharpening data set and performing data preprocessing to obtain image data of each sample, including a high spatial resolution panchromatic image Pan, a low spatial resolution multispectral image ms and a reference image gt, wherein the reference image gt is a high spatial resolution multispectral image; Step 2, forward degeneration of the conditional diffusion model based on the constructed Markov chain of length T steps, stepwise degenerating the reference map gt to the ms map, obtaining the noise map of each time step wherein the time step ; Step 3, constructing a fusion image prediction network deep model and training model parameters; The fusion image prediction network deep model comprises a double-branch encoder, a single-branch decoder, a feature fusion module and a local feature embedding global feature extraction module LEGM; The dual-branch encoder comprises a fusion branch and a noise branch, wherein the fusion branch takes the Pan map and the ms map as conditional inputs, and is used for extracting spatial and spectral features of the image; the noise branch takes the noise map and the time step as inputs, and is used for extracting time-embedded multispectral features; wherein the fusion branch comprises a plurality of cascaded first encoding units, the noise branch comprises a plurality of cascaded second encoding units, the number of the first and second encoding units is the same; and the input of each second encoding unit further comprises the time step The spatially aligned modulation tensor is generated through a time-conditioned modulation mechanism module. The feature fusion module is used for performing feature fusion on the output features of the fusion branch and the noise branch to obtain first fusion features and input the single-branch decoder; The input of the LEGM module comprises the Pan image and the ms image, and the second fusion features are extracted by embedding local features into global features and inputting the single-branch decoder; The single-branch decoder is used for decoding and generating a high spatial resolution multispectral image, and comprises a plurality of cascaded decoding units, the number of the decoding units being consistent with the number of the first / second encoding units; a jump connection is arranged between each decoding unit and the corresponding first encoding unit to receive the first encoding features of the same scale input by the first encoding unit; When training the fused image prediction network deep model, the input includes a Pan image and a ms image of a sample, and an arbitrary time and a corresponding noise map ; the model output is a predicted image of the reference image gt Step 4, generating a high spatial resolution multispectral image of a target image pair based on the trained fusion image prediction network deep model; The target image pair comprises a Pan image and an ms image with the same image size; The initial value of the time step t is set to T, which is decreased to 1 round by round, and the high spatial resolution multispectral images of the target image pairs are generated after T rounds of iteration; wherein the initial value of the noise image input into the noise branch is the ms image, and the noise image input from the time step T-1 is the high spatial resolution multispectral image output by the single-branch decoder in the last round.

2. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, In step 1, the input panchromatic sharpening data set comprises a Pan image and a high spatial resolution multispectral image HRMS; The data preprocessing comprises: cropping the Pan image and the HRMS image to obtain the Pan image and the HRMS image with the same image size; Then, the HRMS image is filtered and down-sampled to generate a low-resolution multispectral image LRMS, and the LRMS is up-sampled to obtain a low spatial resolution multispectral image ms with the same size as the Pan image; and the cropped HRMS image is taken as the reference image gt.

3. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, In step 2, a hyperparameter controlling the noise variance is introduced and displacement sequences determining the noise schedule in the forward degeneration process, specifically comprising: displacement sequence contains a noise coefficient at each time step , denotes the displacement at time step , and the noise coefficient at time step 1 ; when t > 1, ; Based on hyperparameters Boundary conditions for time step 1 are selected are selected According to the formula the corresponding is calculated again according to the noise coefficient of the current time step ; wherein the auxiliary parameter , the auxiliary parameter wherein, is a preset hyper-parameter.

4. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 3, wherein, Displacement of time step 1 The boundary conditions are set as follows: .

5. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, The fusion branch sequentially comprises: a splicing layer, a 3*3 convolution layer, and The first encoding units are cascaded, and one downsampling layer is arranged between adjacent first encoding units to adjust the size of the first encoding features output by the first encoding units. ; The first encoding unit is a residual block of 3 cascaded residual blocks, wherein each residual block comprises two first basic modules, and a residual connection is arranged between the output of the second first basic module and the input of the residual block; wherein the first basic module comprises, in sequence, a Swish activation function, a dropout layer Dropout and a deep separable convolution layer.

6. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, The noise branch comprises, in sequence, a 3x3 convolution layer, and The second encoding unit is cascaded with one down-sampling layer arranged between adjacent first encoding units, for adjusting the size of the second encoding features output by the second encoding unit; wherein, ; The second encoding unit is a first time embedding residual block module of 3 cascaded first time embedding residual blocks, wherein each first time embedding residual block comprises two second basic modules and a time condition modulation mechanism module; the input of the second second basic module is the sum fusion of the output of the first second basic module and the output of the time condition modulation mechanism module, and a residual connection is arranged between the output of the second second basic module and the input of the first time embedding residual block module; The second basic module comprises, in sequence, a Swish activation function, a dropout layer, and a depth separable convolution layer.

7. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, The single-branch decoder sequentially comprises: a decoding unit and a 1x1 convolution layer; wherein the input of each decoding unit comprises a time step t; The first decoding unit further comprises a concatenation layer configured to concatenate the output of the feature fusion module and the first encoded feature of the last first encoding unit of the fusion branch, and take the result as the input of the first decoding unit. to the second decoding to the adjacent decoding units are sequentially provided with an adding layer, an upsampling layer and a splicing layer; wherein the adding layer is configured to add and fuse the second fusion feature output by the LEGM module and the decoding feature output by the previous decoding unit to obtain a decoding fusion feature; the upsampling layer is configured to adjust the size of the decoding fusion feature of the current stage; and the splicing layer is configured to splice the output of the upsampling layer and the same-scale first encoding feature adjusted by the subsequent downsampling layer of the first encoding feature output by the corresponding first encoding unit, and then take the spliced result as the input of the next decoding unit. The input of the 1x1 convolution layer is the sum of the decoded feature output by the last decoding unit and the second fusion feature. Each decoding unit comprises three cascaded second time embedding residual block modules. The input of the second third basic module is the sum of the output of the first third basic module and the output of the time condition modulation mechanism module. The feature fusion module sequentially comprises a splicing layer and two continuous dynamic shape convolution residual blocks; and the input of each continuous dynamic shape convolution residual block further comprises a time step The space-aligned modulation tensor is generated by a time-conditioned modulation mechanism module; wherein the continuous dynamic shape convolution residual block comprises two fourth base modules and a time-conditioned modulation mechanism module; the input of the second fourth base module is the sum fusion of the output of the first fourth base module and the output of the time-conditioned modulation mechanism module, and a residual connection is arranged between the output of the second fourth base module and the input of the continuous dynamic shape convolution residual block; the fourth base module is: replacing the depth separable convolution layer in the first base module with a dynamic shape convolution layer.

9. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, 8. The hybrid convolution and controllable noise collaborative diffusion model remote sensing panchromatic sharpening method of claim 1, wherein 10. The hybrid convolution and controllable noise collaborative diffusion model panchromatic sharpening method of claim 1, wherein, The input of the LEGM module is set as the output of the concatenation layer of the fusion branch, and the feature is extracted by a convolution layer and normalized, and then the obtained feature is sent into two parallel branches for global feature and local feature extraction: the global feature is extracted by using a multi-head attention mechanism, and the local feature is extracted by using a depth separable convolution; after the global feature and the local feature are added and fused, the output of the multi-layer perception is added and fused with the input of the LEGM module through a residual connection, to obtain the output of the LEGM module. The time condition modulation mechanism module comprises, in sequence, a time encoder and a multi-layer perception. The processing procedure of the time encoder comprises: according to the current input time step obtaining the corresponding noise coefficient ; performing multi-dimensional vector representation on the noise coefficient to generate a spatially aligned modulation tensor.

Images