Remote sensing image pan-sharpening method and terminal

Abstract

The present invention relates to the field of image processing technology, and discloses a remote sensing image full-color sharpening method and terminal. The method obtains an image to be fused, including a multispectral image and a full-color image; inputs the image to be fused into a pre-built and qualified diffusion model, and outputs a prediction of the high-resolution multispectral image to complete full-color sharpening; the diffusion model includes: a time step embedding module, a noise scheduling module, and a denoising network. The denoising network is used to perform modal fusion based on the extracted common and difference features to obtain conditional information after performing spatial feature extraction and time-frequency domain feature extraction on the image to be fused, and based on the conditional information and the time token, use an attention mechanism to reconstruct and restore the noisy image or the standard Gaussian noise image to generate a high-resolution multispectral image. While improving the full-color sharpening effect, the present invention reduces the inference time overhead required by the previous diffusion model.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, in particular to a remote sensing image full color sharpening method and a computer terminal applying the method. Background Art

[0002] Satellite remote sensing imagery plays a vital role in the development of modern society. However, due to current technological limitations, it is difficult to directly obtain high-resolution multispectral imagery from remote sensing satellites. Therefore, by equipping satellites with two different types of sensors, we can obtain panchromatic and multispectral imagery, respectively. Panchromatic imagery provides fine spatial detail, while multispectral imagery preserves important spectral characteristics. By fusing the complementary information of these two images, pan-sharpening can produce an image that combines high spatial resolution with spectral information, thereby improving image readability and analytical effectiveness in subsequent remote sensing applications.

[0003] In the actual imaging process, due to technical limitations, remote sensing image acquisition has the following problems: multispectral images have low spatial resolution and blurred details; panchromatic images have only a single band and cannot reflect the spectral information in the actual physical world. To address these problems, there are currently two main solutions:

[0004] (1) Methods based on physical rules, such as component replacement method, variational optimization method and multi-resolution analysis method, etc. Although these methods have clear physical interpretability, their core defect is that they rely on manually designed feature extraction rules, resulting in insufficient feature representation capabilities of images.

[0005] (2) Methods based on deep learning, such as PNN, Panformer and PanDiff, etc. These models are based on convolutional neural networks, Transformer architecture or diffusion models. Through the powerful feature representation and nonlinear modeling capabilities of neural networks, they can automatically learn the complex mapping relationship between multispectral images and full-color images from large-scale data without manually designing feature extraction and fusion rules, which effectively promotes image fusion. However, methods based on deep learning also have obvious defects: 1) Although simple convolutional neural networks successfully capture spatial detail features through local receptive fields, their inherent inductive bias limits the model's global modeling ability of images; 2) Transformer architecture can model the relationship between various parts of an image through a global self-attention mechanism, but because it divides the image into multiple fixed-size image blocks, it is difficult for the model to maintain good robustness when facing noise or low-quality data in the image; 3) Diffusion models mostly use U-Net convolutional networks as the backbone network, and most of them require hundreds of sampling steps in the image generation process, resulting in a large model inference time overhead.

[0006] In summary, there is an urgent need for a remote sensing image fusion solution that can balance image generation quality and efficiency. Summary of the Invention

[0007] To solve the technical problems existing in the prior art, the present invention provides a remote sensing image pan-sharpening method and terminal. The present invention improves the pan-sharpening effect while reducing the inference time overhead required by the previous diffusion model.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] The present invention discloses a remote sensing image panchromatic sharpening method, comprising:

[0010] Acquire images to be fused, including multispectral images and panchromatic images;

[0011] The image to be fused is input into a pre-built and trained diffusion model, which outputs a prediction of the high-resolution multispectral image to complete pan-sharpening. The diffusion model includes:

[0012] The time step embedding module is used to encode time steps into vectors and convert them into time tokens;

[0013] The noise scheduling module is used to generate a set of time-varying noise parameters to control the noise intensity added at each time step, thereby forming a noisy image;

[0014] The denoising network is used to perform modal fusion based on the extracted common and differential features to obtain conditional information after respectively performing spatial feature extraction and time-frequency domain feature extraction on the image to be fused, and to reconstruct and restore the noisy image or the standard Gaussian noise image based on the conditional information and the time token using an attention mechanism to generate a high-resolution multispectral image.

[0015] As a further improvement of the above solution, the denoising network includes:

[0016] The serialization module is used to divide the image to be fused or the noisy image into small blocks and add position information to form a token sequence;

[0017] A reassembly module to reassemble token sequences into images;

[0018] A spatial feature extraction module is used to extract multi-scale spatial features of the image to be fused using a spatial pyramid;

[0019] The time-frequency domain feature extraction module is used to extract the main components of the image to be fused and the high-frequency features in different directions through wavelet transform;

[0020] The modality fusion module is used to extract the differences and common features between different images to be fused, fuse the extracted features, and generate the modality fusion result;

[0021] A reconstruction module is used to generate conditional injection parameters according to the modal fusion result, so as to restore the noisy image to a high-resolution multispectral image through self-attention and cross-attention mechanisms.

[0022] As a further improvement to the above scheme, the training process of the diffusion model includes:

[0023] Forward diffusion stage: Gaussian noise is added to the original image, i.e., the target high-resolution multispectral image in the training set, through a predefined noise scheduling strategy to form a noisy image;

[0024] Time step mapping stage: randomly introduce the conditional information of a time step, convert the time step encoding into a time token, so that the denoising network can perform accurate denoising based on the current time step;

[0025] Denoising network training phase: taking the noisy image as input, the denoising network is trained to predict the original image;

[0026] The testing process of the diffusion model includes:

[0027] Initialization sampling stage: Generate a standard Gaussian noise image without any semantic meaning;

[0028] Implicit sampling stage: The denoising network makes predictions based on the original image. Based on a deterministic reverse formula path, the original image is restored in a sequence of sub-time steps to obtain the prediction result. The denoising network is optimized based on the L1 loss between each prediction result of the denoising network and the original image.

[0029] As a further improvement to the above solution, the process of the serialization module forming the token sequence includes:

[0030] Position information calculation: Sine and cosine position encoding is used to calculate the spatial information at different positions in the image. The expression is as follows:

[0031] ;

[0032] Where, POS Indicates the position of the current token; p It is a self-learning parameter used to adjust the spatial information strength of the position; d is the dimension of the encoding vector; q is the dimension index, indicating the first dimension of the current position encoding vector q dimensions; Indicates the location POS , in the position encoding vectorq Image position encoding value of dimensions; Indicates remainder;

[0033] Serialize the image: The image is divided into image blocks, thereby converting all image blocks into a vector sequence of fixed dimension ;in, is the set of real numbers, The order is number of channels, height, and width; , For the size of each image block, is the dimension of each image block, ;

[0034] Add position information: Add the above calculated sine and cosine position codes to the vector sequence bit by bit to form a token sequence with position information. The expression is as follows:

[0035] ;

[0036] Where, For images I The corresponding token sequence, Encodes information about the image location.

[0037] As a further improvement to the above solution, the feature extraction process of the spatial feature extraction module includes:

[0038] Construct a three-layer spatial pyramid of an image: The spatial pyramid is constructed from the image I The extracted spatial features of different scales are composed of , use convolution with a step size of 2 for downsampling to extract spatial features of different scales. The expression is as follows:

[0039] ;

[0040] Where, i is the sequence number of the pyramid layer; For images I The obtained space pyramid i Layer space characteristics; Conv (·) is convolution processing; is the set of real numbers, The order is number of channels, height, and width;

[0041] Promoting feature fusion: For multispectral images M and full-color images P The spatial features in the obtained spatial pyramid and , feature fusion is promoted through affine transformation, and the expression is as follows:

[0042] ;

[0043] ;

[0044] Where, and They are and Spatial features that promote feature fusion through affine transformation; is element-wise multiplication;

[0045] Restore the spatial features of different scales of feature maps: and Re-upsample back to the original spatial size through transposed convolution, the expression is as follows:

[0046] ;

[0047] ;

[0048] Where, and They are and The restored spatial features; is the transposed convolution;

[0049] Stitching features: Stitching the restored spatial features of the multispectral image and the panchromatic image respectively to obtain the required spatial features and .

[0050] As a further improvement of the above solution, the feature extraction process of the time-frequency domain feature extraction module includes:

[0051] Wavelet transform: for images I Wavelet transform is used to extract the main components and high-frequency features in different directions. The expression is as follows:

[0052] ;

[0053] Where, represents wavelet transform; Represents high-frequency features in the diagonal direction; Represents the detail component in the vertical line direction; Represents the detail component in the horizontal direction; represents the low-frequency principal component;

[0054] Feature upsampling: For multispectral images MSelect the principal component features, for full-color images P Select high-frequency features and upsample the wavelet transform results using the bicubic interpolation method to restore them to their original spatial size. The expression is as follows:

[0055] ;

[0056] ;

[0057] Where, is the time-frequency domain feature of the upsampled multispectral image; is the time-frequency domain feature of the upsampled full-color image; Represents the low-frequency principal component of the multispectral image obtained by wavelet transform; Represents the high-frequency features in the diagonal direction of the full-color image obtained by wavelet transform; Represents the detail component in the vertical line direction of the full-color image; Represents the detail component of the panchromatic image in the horizontal direction; Indicates bicubic interpolation processing;

[0058] Feature extraction: Multispectral image M and Splice and extract features to convert full-color image P and After performing affine transformation and then extracting features, the expressions are as follows:

[0059] ;

[0060] ;

[0061] Where, To meet the requirements of time-frequency domain characteristics of multispectral images; To meet the requirements of the time-frequency domain characteristics of the full-color image.

[0062] As a further improvement of the above solution, the process of the modal fusion module generating the modal fusion result includes:

[0063] Extract QKV triples: Divide the spatial features or time-frequency domain features of the multispectral image and the panchromatic image into three parts as the query, key, and value of the attention mechanism. The expressions are as follows:

[0064] ;

[0065] ;

[0066] Where, is the spatial feature or time-frequency domain feature of the multispectral image, is the spatial feature or time-frequency domain feature of the full-color image, 、 and From the characteristics Query, Key and Value; 、 and From the characteristics Query, Key and Value; Indicates equal distribution;

[0067] Generate global token: Through multi-layer perceptron and pooling layer and The serialized result and Processing is performed to obtain the global token, the expression is as follows:

[0068] ;

[0069] Where, Represents a global token; represents average pooling; Represents multi-layer perceptron processing;

[0070] Extracting difference features: Using similarity matrix extraction in attention mechanism calculation and Common features, use After subtraction, it is used as the difference feature and adjusted using the linear layer. The expression is as follows:

[0071] ;

[0072] Where, is the adjusted difference characteristic; It is processed by linear layer; Represents the similarity matrix processing in the attention mechanism calculation;

[0073] Extract modal fusion results: Extract the common features twice and add them to the difference features to obtain the complete image features. Then use convolution to perform upsampling to obtain the modal fusion results. The expression is as follows:

[0074] ;

[0075] ;

[0076] ;

[0077] Where, and are the results of extracting common features twice, is the modal fusion result.

[0078] As a further improvement of the above solution, the process of generating a high-resolution spectral image by the reconstruction module includes:

[0079] Serialization: Serialize the noisy image N into a token sequence ;

[0080] Generate conditional injection parameters: By combining the modal fusion results The conditional injection parameters are evenly divided and expressed as follows:

[0081] ;

[0082] Wherein, the reconstruction module is equipped with reorganization normalization; Used for scaling in recombinant normalization; Used for bias in reorganization normalization; For gate control; Indicates the key and value features provided by the conditions in the cross attention;

[0083] Image denoising: Use the self-attention mechanism to reconstruct the image of the token sequence. The expression is as follows:

[0084] ;

[0085] ;

[0086] Where, is the k-th level processing result of the token sequence; Normalization of the representation layer; Represents the self-attention mechanism; is the time token corresponding to time step t, t∈[1,2,…,T], T is the total number of time steps;

[0087] Using the cross attention mechanism Strengthen the conditional information, the expression is as follows:

[0088] ;

[0089] ;

[0090] ;

[0091] Where, and The Key and Value from the condition respectively; It is a cross attention mechanism;

[0092] Using multi-layer perceptron After adjustment, the expression is as follows:

[0093] ;

[0094] ;

[0095] Image reconstruction: Repeat the above steps several times and use the linear layer to reconstruct the final Downsampling is performed and then compared with the multispectral image M Add to generate high-resolution multispectral images :

[0096] ;

[0097] Where, Represents linear layer processing; Indicates reorganization.

[0098] As a further improvement of the above solution, the process of generating time tokens by the time step embedding module includes:

[0099] Generate time position encoding: generate a discrete time step range [1, 2, ..., T] based on the total number of time steps T;

[0100] The time step t∈[1,2,…,T] is encoded using the sine and cosine functions, as shown below:

[0101] ;

[0102] Where, is the encoding of time step t; dim is the encoding dimension; m is the dimension index; Indicates remainder;

[0103] Use a multi-layer perceptron to convert the encoding of time step t into a time token, as shown below:

[0104] ;

[0105] Where, is the time token corresponding to time step t; Represents multi-layer perceptron processing;

[0106] The process of forming a noisy image by the noise scheduling module includes:

[0107] The accumulation degree of noise intensity is controlled by the cosine function, and the expression is as follows:

[0108] ;

[0109] Where, represents the cumulative retention factor from time step 0 to time step t; is a smoothing parameter;

[0110] During the model training phase, noise is added to the original image to generate a noisy image. The noisy image expression at time t is as follows:

[0111] ;

[0112] Where, represents the noisy image corresponding to time step t; represents the original image; is Gaussian noise;

[0113] When the actual model is sampled and predicted, a sub-time step sequence is sampled from a time step sequence with a total time step number of T. , s is the total number of sub-time steps, so the accelerated sampling is performed according to the following formula:

[0114] ;

[0115] Where, , represents the denoising network, represents the sub-time step of the current sampling, Indicates the sub-time step in the sampling phase The noisy image under c For multispectral images and panchromatic images, For the denoising network The predicted value of For sub-time steps The noise intensity under .

[0116] The present invention also discloses a computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the remote sensing image full-color sharpening method described above are implemented.

[0117] Compared with the prior art, the present invention has the following beneficial effects:

[0118] 1. The remote sensing image full-color sharpening method disclosed in this invention successfully compresses the sampling time step required for the diffusion model inference process through a diffusion model and a denoising network based on the Transformer architecture, thereby improving the full-color sharpening effect while reducing the inference time overhead required by the previous diffusion model.

[0119] 2. The spatial feature extraction module proposed in the present invention uses a spatial pyramid to extract multi-scale features of the image, and uses adaptive modulation to promote feature fusion; the time-frequency domain feature extraction module uses wavelet transform to extract the main components of the image and high-frequency features in different directions, and uses the local receptive field of convolution to learn image details, so as to fully extract the features contained in the image in the spatial and time-frequency domains.

[0120] 3. The modal fusion module proposed in the present invention calculates the common features contained in images of different modalities through a similarity matrix, then subtracts the common features from the features extracted from images of different modalities to obtain the difference features of the images. Finally, the detail features of the image are enhanced by superimposing the difference features on the complete features, thereby effectively promoting the feature complementarity of images of different modalities.

[0121] 4. The image generation results of this remote sensing image full-color sharpening method can be applied in many fields, such as mineral exploration, environmental testing, etc. The processing process is automated, no human intervention is required, and the running time is low, so it has strong applicability.

[0122] 5. The computer terminal and storage medium disclosed in the present invention can produce the same beneficial effects by applying the above method, and will not be described in detail. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] Figure 1 Flowchart of the remote sensing image pan-sharpening method in Example 1 of the present invention.

[0124] Figure 2 This is a flow chart of the diffusion model training method in Example 1 of the present invention.

[0125] Figure 3 This is a diagram of the overall architecture of the denoising network in Example 1 of the present invention.

[0126] Figure 4 for Figure 3 Schematic diagram of the principle of the conditional injection block (the upper part of the figure is the reconstruction module).

[0127] Figure 5 for Figure 4 Schematic diagram of the principle of recombinant normalization.

[0128] Figure 6 for Figure 4 Schematic diagram of the principle of mid-space feature extraction.

[0129] Figure 7 for Figure 4 Schematic diagram of the principle of feature extraction in the time-frequency domain.

[0130] Figure 8 for Figure 4 Schematic diagram of the principle of mesomodal fusion.

[0131] Figure 9 This is a comparison chart of the effects of the remote sensing image full color sharpening method in Example 1 of the present invention and the mainstream method.

[0132] Figure 10 This is a schematic diagram of the structure of a computer terminal in Example 2 of the present invention. DETAILED DESCRIPTION

[0133] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0134] Example 1

[0135] See also Figure 1 This embodiment provides a remote sensing image pan-sharpening method, comprising:

[0136] Acquire images to be fused, including multispectral images and panchromatic images;

[0137] The image to be fused is input into a pre-built and trained diffusion model, which outputs a prediction of the high-resolution multispectral image to complete panchromatic sharpening.

[0138] See also Figure 2 In this embodiment, before training the diffusion model, existing data can be used to acquire low-resolution multispectral and panchromatic images. These images are cropped to a predetermined size to construct a dataset, which is then proportionally divided into training and test sets. During training, samples and time steps are randomly selected to generate corresponding noisy images. The model predicts the initial image based on the noisy image, time step, and conditional information, and trains until the model converges. The trained diffusion model is then used to make predictions for a given set of images through implicit sampling.

[0139] The diffusion model of the present invention comprises: a time step embedding module, a noise scheduling module and a denoising network.

[0140] The time step embedding module is used to encode the time step into a vector and convert it into a time token. The specific process includes the following steps, namely S11~S12.

[0141] S11. Generate time position code:

[0142] Generate a discrete time step range [1,2,…,T] based on the total number of time steps T.

[0143] The time step t∈[1,2,…,T] is encoded using the sine and cosine functions, as shown below:

[0144] ;

[0145] Where, is the encoding of time step t; dim is the encoding dimension; m is the dimension index; Indicates remainder.

[0146] S12. Use a multi-layer perceptron to convert the encoding of time step t into a time token, as shown below:

[0147] ;

[0148] Where, is the time token corresponding to time step t; Represents multilayer perceptron processing.

[0149] The noise scheduling module is used to generate a set of noise parameters that vary with time and control the noise intensity added at each time step to form a noisy image.

[0150] The process of forming a noisy image by the noise scheduling module includes the following steps, namely S21-S22.

[0151] S21. The accumulation of noise intensity is controlled by the cosine function. The expression is as follows:

[0152] ;

[0153] Where, represents the cumulative retention factor from time step 0 to time step t; is a smoothing parameter used to avoid the noise being too small in the initial stage.

[0154] S22. At each time step, noise is added to the original image to generate a noisy image. The expression is as follows:

[0155] ;

[0156] Where, represents the noisy image corresponding to time step t; represents the original image; is Gaussian noise.

[0157] The denoising network is used to perform modal fusion based on the extracted common and difference features to obtain conditional information after performing spatial feature extraction and time-frequency domain feature extraction on the image to be fused, and to reconstruct and restore the noisy image or standard Gaussian noise image based on the conditional information and the time token using an attention mechanism to generate a high-resolution multispectral image.

[0158] See also Figures 3 to 8 ,The denoising network includes: serialization module, recombination module, spatial feature extraction module, time-frequency domain feature extraction module, modal fusion module and reconstruction module.

[0159] The serialization module is used to divide the image to be fused or the noisy image into small blocks and add position information to form a token sequence. The specific process includes the following steps, namely S31~S33.

[0160] S31. Position information calculation:

[0161] Sin-cosine position coding is used to calculate the spatial information at different positions in the image. The expression is as follows:

[0162] ;

[0163] Where, POS Indicates the position of the current token; p It is a self-learning parameter used to adjust the spatial information strength of the position; d is the dimension of the encoding vector; q is the dimension index, indicating the first dimension of the current position encoding vector q dimensions; Indicates the location POS , in the position encoding vector q Image position encoding value of dimensions; Indicates remainder.

[0164] S32. Serialize the image:

[0165] The image The image is divided into image blocks, thereby converting all image blocks into a vector sequence of fixed dimension ;in, is the set of real numbers, The order is number of channels, height, and width; , For the size of each image block, is the dimension of each image block, .

[0166] S33. Add location information:

[0167] The above-calculated sine and cosine position codes are added bit by bit to the vector sequence to form a token sequence with position information. The expression is as follows:

[0168] ;

[0169] Where, For images The corresponding token sequence, Encodes information about the image location.

[0170] The reorganization module is used to reorganize the token sequence into an image. Let the token sequence , the known image block size is ,but , each image block dimension , you can Rescaled to .

[0171] The spatial feature extraction module is used to extract multi-scale spatial features of the image to be fused using a spatial pyramid. The specific process includes the following steps, namely S41 to S44.

[0172] S41. Construct a three-layer spatial pyramid of the image:

[0173] The spatial pyramid is constructed from the image I The extracted spatial features of different scales are composed of , use convolution with a step size of 2 for downsampling to extract spatial features of different scales. The expression is as follows:

[0174] ;

[0175] Where, i is the sequence number of the pyramid layer; For the image I The obtained space pyramid i Layer space characteristics; Conv (·) is convolution processing; is the set of real numbers, The order is number of channels, height, and width.

[0176] S42. Promote feature fusion:

[0177] For multispectral images M and full-color images P The spatial features in the obtained spatial pyramid and , feature fusion is promoted through affine transformation, and the expression is as follows:

[0178] ;

[0179] ;

[0180] Where, and They are and Spatial features that promote feature fusion through affine transformation; is element-wise multiplication.

[0181] S43. Restore spatial features of feature maps at different scales:

[0182] The spatial features and Re-upsample back to the original spatial size through transposed convolution, the expression is as follows:

[0183] ;

[0184] ;

[0185] Where, and They are and The restored spatial features; is the transposed convolution.

[0186] S44, splicing features:

[0187] The restored spatial features corresponding to the multispectral image and the panchromatic image are spliced ​​separately to obtain the required spatial features. and .

[0188] The time-frequency domain feature extraction module is used to extract the main components of the image to be fused and the high-frequency features in different directions through wavelet transform. The specific process includes the following steps, namely S51 to S53.

[0189] S51. Wavelet transform:

[0190] For images Wavelet transform is used to extract the main components and high-frequency features in different directions. The expression is as follows:

[0191] ;

[0192] Where, represents wavelet transform; Represents high-frequency features in the diagonal direction; Represents the detail component in the vertical line direction; Represents the detail component in the horizontal direction; Represents the low-frequency principal component.

[0193] S52, feature upsampling:

[0194] For multispectral images M Select the principal component features, for full-color images PSelect high-frequency features and upsample the wavelet transform results using the bicubic interpolation method to restore them to their original spatial size. The expression is as follows:

[0195] ;

[0196] ;

[0197] Where, is the time-frequency domain feature of the upsampled multispectral image; is the time-frequency domain feature of the upsampled full-color image; Represents the low-frequency principal component of the multispectral image obtained by wavelet transform; Represents the high-frequency features in the diagonal direction of the full-color image obtained by wavelet transform; Represents the detail component in the vertical line direction of the full-color image; Represents the detail component of the panchromatic image in the horizontal direction; Indicates bicubic interpolation processing.

[0198] S53, Feature Extraction:

[0199] Multispectral image M and Splice and extract features to convert full-color image P and After performing affine transformation and then extracting features, the expressions are as follows:

[0200] ;

[0201] ;

[0202] Where, To meet the requirements of time-frequency domain characteristics of multispectral images; To meet the requirements of the time-frequency domain characteristics of the full-color image.

[0203] The modality fusion module is used to extract the differences and common features between different images to be fused, fuse the extracted features, and generate a modality fusion result. The specific process includes the following steps, namely S61 to S64.

[0204] S61. Extract QKV triples:

[0205] The spatial features or time-frequency domain features of the multispectral image and the panchromatic image are divided into three parts respectively as the query, key and value (i.e. Query, Key and Value) of the attention mechanism. The expressions are as follows:

[0206] ;

[0207] ;

[0208] Where, is the spatial feature or time-frequency domain feature of the multispectral image, is the spatial feature or time-frequency domain feature of the full-color image, 、 and From the characteristics Query, Key and Value; 、 and From the characteristics Query, Key and Value; Indicates equal distribution.

[0209] S62. Generate a global token:

[0210] Through multi-layer perceptron and pooling layer and The serialized result and Processing is performed to obtain the global token, the expression is as follows:

[0211] ;

[0212] Where, Represents a global token; represents average pooling; Represents multilayer perceptron processing.

[0213] S63, extracting difference features:

[0214] Similarity matrix extraction using attention mechanism calculation and Common features, use After subtraction, it is used as the difference feature and adjusted using the linear layer. The expression is as follows:

[0215] ;

[0216] Where, is the adjusted difference characteristic; It is processed by linear layer; Represents the similarity matrix processing in the attention mechanism calculation.

[0217] S64. Extract modal fusion results:

[0218] Extract the common features twice and add them to the difference features to get the complete image features. Then use convolution to perform upsampling to get the modal fusion result. The expression is as follows:

[0219] ;

[0220] ;

[0221] ;

[0222] Where, and are the results of extracting common features twice, is the modal fusion result.

[0223] The reconstruction module is used to generate conditional injection parameters based on the modal fusion results, thereby restoring the noisy image to a high-resolution multispectral image through self-attention and cross-attention mechanisms. The specific process includes the following steps, namely S71~S74.

[0224] S71. Serialization:

[0225] Serialize the noisy image N into a token sequence , the expression formula is: .

[0226] S72. Generate conditional injection parameters:

[0227] By combining the modal fusion results The conditional injection parameters are evenly divided and expressed as follows:

[0228] ;

[0229] Wherein, the reconstruction module is equipped with reorganization normalization; Used for scaling in recombinant normalization; Used for bias in reorganization normalization; For gate control; Indicates the features of the key and value provided from the condition in the cross attention.

[0230] S73, Image Denoising:

[0231] The self-attention mechanism is used to reconstruct the image of the token sequence. The expression is as follows:

[0232] ;

[0233] ;

[0234] Where, is the k-th level processing result of the token sequence; Normalization of the representation layer; Represents the self-attention mechanism; is the time token corresponding to time step t, t∈[1,2,…,T], and T is the total number of time steps.

[0235] Using the cross attention mechanism Strengthen the conditional information, the expression is as follows:

[0236] ;

[0237] ;

[0238] ;

[0239] Where, and The Key and Value from the condition respectively; It is a cross attention mechanism.

[0240] Using multi-layer perceptron After adjustment, the expression is as follows:

[0241] ;

[0242] ;

[0243] S74, Image Reconstruction:

[0244] Repeat steps S71 to S73 several times, and use the linear layer to adjust the final S73 output. Downsampling is performed and then compared with the multispectral image M Add to generate high-resolution multispectral images :

[0245] ;

[0246] Where, Represents linear layer processing; Indicates reorganization.

[0247] In this embodiment, the training process of the diffusion model includes:

[0248] Forward diffusion stage: Gaussian noise is added to the original image, i.e., the target high-resolution multispectral image in the training set, through a predefined noise scheduling strategy to form a noisy image;

[0249] Time step mapping stage: randomly introduce the conditional information of a time step, convert the time step encoding into a time token, so that the denoising network can perform accurate denoising according to the current time step.

[0250] Denoising network training phase: Take the noisy image as input and train the denoising network to predict the original image.

[0251] The training process is as follows:

[0252] Sample a target image from the dataset And the corresponding images to be fused M, P;

[0253] Randomly sample a random time step ;

[0254] Sampling noise , construct a noisy image;

[0255] High-resolution multispectral imagery predicted using diffusion models ;

[0256] Calculate the L1 loss and backpropagate the training model. The loss expression is as follows:

[0257] ;

[0258] Where, is the mathematical expectation symbol; To calculate the L1 norm.

[0259] The optimizer configuration is as follows:

[0260] The Adam optimizer is used; the initial learning rate is set to 1e-4; the cosine annealing scheduler is used to dynamically adjust the learning rate; and gradient clipping is applied to prevent gradient explosion.

[0261] Training data processing: The input includes multispectral images and panchromatic images, and the images are standardized.

[0262] In addition, the configuration design of the diffusion model includes: the image block size is set to 8; the token dimension is set to 1024; and 12 conditional injection blocks.

[0263] The testing process of the diffusion model includes:

[0264] Initialization sampling stage: Generate a standard Gaussian noise image without any semantic meaning;

[0265] Implicit sampling stage: The denoising network makes predictions based on the original image. Based on a deterministic reverse formula path, the original image is restored in a sequence of sub-time steps to obtain the prediction result. The denoising network is optimized based on the L1 loss between each prediction result of the denoising network and the original image.

[0266] During the model training phase, noise is added to the original image to generate a noisy image. The noisy image expression at time t is as follows:

[0267] ;

[0268] Where, represents the noisy image corresponding to time step t; represents the original image; is Gaussian noise.

[0269] During the model sampling phase, an inverse process can be obtained that is determined only by the actual noise added:

[0270] ;

[0271] Where, represents the noisy image corresponding to time step t-1. The noisy image here refers to the image restored midway during the sampling phase; represents the original image; for Gaussian noise added at the time step.

[0272] The above formula can be transformed into:

[0273] ;

[0274] This shows that for the target original image , the sampling path generated is uniquely determined by the initially generated random Gaussian noise image.

[0275] Therefore, the actual model sampling prediction, that is, the testing process is as follows:

[0276] Generate an initial Gaussian noise image and use it as ;

[0277] Sample a sub-time step sequence from a time step sequence with a total time step number T , perform accelerated sampling according to the following formula:

[0278] ;

[0279] Where, , represents the denoising network, represents the sub-time step of the current sampling, Indicates the sub-time step in the sampling phase The noisy image under c For multispectral images and panchromatic images, For the denoising network The predicted value of For sub-time steps The noise intensity under .

[0280] In order to verify the effectiveness of the remote sensing image pan-sharpening method proposed in the present invention, this embodiment conducted a large number of experiments on multiple data sets, and the experimental results are as follows.

[0281] 1. Low-resolution remote sensing image full color sharpening experiment

[0282] This example conducts experiments on three datasets: WorldView-II, Gaofen-2, and WorldView-III, and compares them with the current mainstream full-color sharpening methods (SFIM[1], Brovey[2], IHS[3], GS[4], GFPCA[5], PanNet[6], MSDCNN[7], SRPPNN[8], INNformer[9], SFINet

[10] , MSDDN

[11] , PanFlowNet

[12] , Pan-Mamba

[13] ).

[0283] The above mainstream methods are:

[0284] [1] J. Liu, “Smoothing filter-based intensity modulation: A spectralpreserve image fusion technique for improving spatial details,” International Journal of remote sensing, vol. 21, no. 18, pp. 3461–3472, 2000.

[0285] [2] AR Gillespie, AB Kahle, and RE Walker, “Colorenhancement of highly correlated images. ii. channel ratio and “chromaticity” transformation techniques,” Remote Sensing of Environment, vol. 22, no. 3, pp. 343–365, 1987.

[0286] [3] R. Haydn, “Application of the ihs color transform to theprocessing of multisensor data and image enhancement,” in Proc. of theInternational Symposium on Remote Sensing of Arid and SemiArid Lands, Cairo,Egypt, 1982, 1982.

[0287] [4] B. Aiazzi, S. Baronti, and M. Selva, “Improving componentsubstitution pansharpening through multivariate regression of ms + pan data,”IEEE Transactions on Geoscience and Remote Sensing, vol. 45, no. 10, pp.3230–3239, 2007.

[0288] [5] W. Liao, X. Huang, F. Van Coillie, G. Thoonen, A. Pizurica, P.Scheunders, and W. Philips, “Two-stage fusion of thermal hyperspectral andvisible rgb image by pca and guided filter,” in 2015 7th Workshop onHyperspectral Image and Signal Processing: Evolution in Remote Sensing(WHISPERS). Ieee, 2015, pp. 1–4.

[0289] [6] J. Zhong, B. Yang, G. Huang, F. Zhong, and Z. Chen, “Remotesensing image fusion with convolutional neural network,” Sensing and Imaging,vol. 17, pp. 1–16, 2016.

[0290] [7] Q. Yuan, Y. Wei, X. Meng, H. Shen, and L. Zhang, “A multiscaleand multidepth convolutional neural network for remote sensing imagery pan-sharpening,” IEEE Journal of Selected Topics in Applied Earth Observationsand Remote Sensing, vol. 11, no. 3, pp. 978–989, 2018.

[0291] [8] J. Cai and B. Huang, “Super-resolution-guided progressivepansharpening based on a deep convolutional neural network,” IEEETransactions on Geoscience and Remote Sensing, vol. 59, no. 6, pp. 5206–5220,2020.

[0292] [9] M. Zhou, J. Huang, Y. Fang, X. Fu, and A. Liu, “Pan-sharpeningwith customized transformer and invertible neural network,” in Proceedings ofthe AAAI conference on artificial intelligence, vol. 36, no. 3, 2022, pp.3553–3561.

[0293]

[10] M. Zhou, J. Huang, K. Yan, H. Yu, X. Fu, A. Liu, X. Wei, and F.Zhao, “Spatial-frequency domain information integration for pan-sharpening,”in European conference on computer vision. Springer, 2022, pp. 274–291.

[0294]

[11] X. He, K. Yan, J. Zhang, R. Li, C. Xie, M. Zhou, and D. Hong,“Multiscale dual-domain guidance network for pan-sharpening,” IEEETransactions on Geoscience and Remote Sensing, vol. 61, pp. 1–13, 2023.

[0295]

[12] G. Yang, X. Cao, W. Xiao, M. Zhou, A. Liu, X. Chen, and D. Meng,“Panflownet: A flow-based deep network for pan-sharpening,” in Proceedings ofthe IEEE / CVF International Conference on Computer Vision, 2023, pp. 16 857–16867.

[0296]

[13] X. He, K. Cao, J. Zhang, K. Yan, Y. Wang, R. Li, C. Xie, D.Hong, and M. Zhou, “Pan-mamba: Effective pan-sharpening with state spacemodel,” Information Fusion, vol. 115, p. 102779, 2025.

[0297] Table 1 shows the quantitative comparison results. The bold and underlined lines represent the best and suboptimal results, respectively. “Ours-2” and “Ours-25” represent the results of the two-step and 25-step sampling methods, respectively. The evaluation indicators include:

[0298] Peak signal-to-noise ratio (PSNR): measures the difference between the reconstructed image and the target image;

[0299] Structural Similarity (SSIM): Evaluates the similarity between images based on brightness, contrast, and structure;

[0300] Spectral Angle Mapping ( ): Evaluates the angular difference between the spectral vectors between images;

[0301] Comprehensive relative global error ( ): Measures the relative error between the reconstructed image and the target image in different bands.

[0302] Table 1: Quantitative comparison with existing methods on the task of pan-sharpening low-resolution remote sensing images

[0303] ;

[0304] As can be seen in Table 1, the proposed method achieves optimal or suboptimal results across multiple evaluation metrics. Even when the number of sampling steps is reduced, no significant decrease in these metrics is observed. This demonstrates that the proposed method can achieve high-quality fusion while using fewer sampling steps.

[0305] 2. Full-resolution remote sensing image panchromatic sharpening experiment

[0306] This example was experimented on the full-resolution WorldView-II dataset and compared with the current mainstream pan-sharpening method. Table 2 shows the quantitative comparison results. The evaluation indicators include:

[0307] Spectral distortion ( ): Measures the distortion between the reconstructed image and the original multispectral image in the spectral dimension;

[0308] Spatial distortion ( ): Measures the consistency of spatial details between the reconstructed image and the full-color image;

[0309] No-reference image fusion quality index ( ): Comprehensive evaluation of fusion quality.

[0310] Table 2: Quantitative comparison with existing methods on the full-resolution WorldView-II dataset

[0311] ;

[0312] As can be seen from Table 2, in the full-resolution remote sensing image pan-sharpening task, the method of the present invention is superior to other advanced methods in non-reference quality indicators, and the prediction effect of the model does not decrease after reducing the number of sampling steps, which shows that the present invention has good application potential.

[0313] See also Figure 9 The upper half of the figure shows the visualization of the synthesized high-resolution multispectral image, while the lower half shows the distribution of the mean squared error between the synthesized result and the true value. Brighter areas indicate larger differences. It can be seen that the image generated by the remote sensing image pan-sharpening method of the present invention is closest to the true image, with better overall detail and spectral accuracy.

[0314] Example 2

[0315] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the remote sensing image full-color sharpening method described in Example 1 are implemented.

[0316] like Figure 10 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to the at least one processor 101. The specific connection medium between the processor 101 and the memory 102 is not limited in this embodiment. Figure 10 In the example, the processor 101 and the memory 102 are connected via the bus 100. Figure 10 The bus 100 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by only one thick line, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 101 can also be called a controller, and there is no limitation on the name.

[0317] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101 , and the at least one processor 101 can perform the aforementioned method by executing the instructions stored in the memory 102 .

[0318] Among them, the processor 101 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire control device, and monitor the device as a whole by running or executing instructions stored in the memory 102 and calling data stored in the memory 102, the various functions of the device and processing data.

[0319] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0320] The processor 101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the remote sensing image pan-sharpening method disclosed in Example 1 can be directly implemented and executed by a hardware processor, or by a combination of hardware and software modules in the processor 101.

[0321] The memory 102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (RAM), a static random access memory (SRAM), a programmable read-only memory (PROM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a magnetic disk, an optical disk, and the like. The memory 102 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 102 in this embodiment can also be a circuit or any other device that can implement a storage function, for storing program instructions and / or data.

[0322] By programming the processor 101, the code corresponding to the security verification method described in the above embodiment can be fixed into the chip, so that the chip can execute the security verification method when it is running. Figure 1How to design and program the processor 101 is a technique well known to those skilled in the art and will not be described in detail here.

[0323] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A remote sensing image pan-sharpening method, characterized in that: include: Acquire images to be fused, including multispectral images and panchromatic images; The image to be fused is input into a pre-built and trained diffusion model, which outputs a prediction of the high-resolution multispectral image to complete pan-sharpening. The diffusion model includes: The time step embedding module is used to encode time steps into vectors and convert them into time tokens; The noise scheduling module is used to generate a set of time-varying noise parameters to control the noise intensity added at each time step, thereby forming a noisy image; A denoising network is configured to, after extracting spatial features and time-frequency domain features from the image to be fused, perform modal fusion based on the extracted common and differential features to obtain conditional information, and reconstruct and restore the noisy image or the standard Gaussian noise image using an attention mechanism based on the conditional information and the time token to generate a high-resolution multispectral image; the denoising network comprises: The serialization module is used to divide the image to be fused or the noisy image into small blocks and add position information to form a token sequence; A reassembly module to reassemble token sequences into images; A spatial feature extraction module is used to extract multi-scale spatial features of the image to be fused using a spatial pyramid; The time-frequency domain feature extraction module is used to extract the main components of the image to be fused and the high-frequency features in different directions through wavelet transform; The modality fusion module is used to extract the differences and common features between different images to be fused, fuse the extracted features, and generate the modality fusion result; A reconstruction module is used to generate conditional injection parameters according to the modal fusion result, so as to restore the noisy image to a high-resolution multispectral image through self-attention and cross-attention mechanisms.

2. The remote sensing image pan-sharpening method according to claim 1, characterized in that: The training process of the diffusion model includes: Forward diffusion stage: Gaussian noise is added to the original image, i.e., the target high-resolution multispectral image in the training set, through a predefined noise scheduling strategy to form a noisy image; Time step mapping stage: randomly introduce the conditional information of a time step, convert the time step encoding into a time token, so that the denoising network can perform accurate denoising based on the current time step; Denoising network training phase: taking the noisy image as input, the denoising network is trained to predict the original image; The testing process of the diffusion model includes: Initialization sampling stage: Generate a standard Gaussian noise image without any semantic meaning; Implicit sampling stage: The denoising network makes predictions based on the original image. Based on a deterministic reverse formula path, the original image is restored in a sequence of sub-time steps to obtain the prediction result. The denoising network is optimized based on the L1 loss between each prediction result of the denoising network and the original image.

3. The remote sensing image pan-sharpening method according to claim 1, characterized in that: The process of forming the token sequence by the serialization module includes: Position information calculation: Sine and cosine position encoding is used to calculate the spatial information at different positions in the image. The expression is as follows: Where pos represents the position of the current token; p is a self-learning parameter used to adjust the spatial information strength of the position; d is the dimension of the encoding vector; q is the dimension index, indicating the qth dimension of the encoding vector at the current position; PE(pos,q) represents the image position encoding value at position pos, in the qth dimension of the position encoding vector; % represents the remainder; Serialize the image: The image is divided into n image blocks, so that all image blocks are converted into a vector sequence of fixed dimension. in, is a set of real numbers, C, H, and W are the number of channels, height, and width respectively; P is the size of each image block, D is the dimension of each image block, D = P 2 C; Add position information: Add the above calculated sine and cosine position codes to the vector sequence bit by bit to form a token sequence with position information. The expression is as follows: x I =x patch +PE Where x I is the token sequence corresponding to image I, and PE is the image position encoding information.

4. The remote sensing image pan-sharpening method according to claim 1, characterized in that: The feature extraction process of the spatial feature extraction module includes: Construct a three-layer spatial pyramid of the image: The spatial pyramid consists of several spatial features of different scales extracted from the image I. Use convolution with a stride of 2 for downsampling to extract spatial features of different scales. The expression is as follows: Where i is the number of the pyramid layer; F i is the spatial feature of the i-th layer of the spatial pyramid obtained from image I; Conv(·) is the convolution process; is a set of real numbers, C, H, and W are the number of channels, height, and width respectively; Promote feature fusion: For the spatial features F in the spatial pyramid obtained from the multispectral image M and the panchromatic image P M,i and F P,i , feature fusion is promoted through affine transformation, and the expression is as follows: F M,i 1 =Conv(F P,i )⊙F M,i +Conv(F P,i ) f P,i 1 =Conv(F M,i )⊙F P,i +Conv(F M,i ) Where, F M,i 1 and F P,i 1 F M,i and F P,i The spatial features that promote feature fusion through affine transformation; ⊙ is element-wise multiplication; Restore the spatial features of different scales of the feature map: Restore the spatial features F M,i 1 and F P,i 1 Re-upsample back to the original spatial size through transposed convolution, the expression is as follows: F M,i 2 =DeConv(F M,i 1 ) F P,i 2 =DeConv(F P,i 1 ) Where, F M,i 2 and F P,i 2 F M,i 1 and F P,i 1 The restored spatial features; DeConv(·) is the transposed convolution; Stitching features: Stitching the restored spatial features of the multispectral image and the panchromatic image respectively to obtain the required spatial features and 5. The remote sensing image pan-sharpening method according to claim 4, characterized in that: The feature extraction process of the time-frequency domain feature extraction module includes: Wavelet transform: For image I, wavelet transform is used to extract the main components and high-frequency features in different directions. The expression is as follows: [HH,HL,LH,LL]=db1(I) Where, db1(·) represents wavelet transform; Represents high-frequency features in the diagonal direction; Represents the detail component in the vertical line direction; Represents the detail component in the horizontal direction; represents the low-frequency principal component; Feature upsampling: For the multispectral image M, the principal component features are selected; for the panchromatic image P, the high-frequency features are selected. The wavelet transform result is upsampled by the bicubic interpolation method to restore it to the original spatial size. The expression is as follows: F M ′=bicubic([LL M ]) F P ′=bicubic([HH P ,HL P ,LH P ]) Where, F M ′ is the time-frequency domain feature of the multispectral image after upsampling; F P ′ is the time-frequency domain feature of the full-color image after upsampling; LL M Represents the low-frequency principal component of the multispectral image obtained by wavelet transform; HH P Represents the high-frequency features in the diagonal direction of the full-color image obtained by wavelet transform; HL P Represents the detail component in the vertical line direction of the full-color image; LH P Represents the detail component of the horizontal line direction of the full-color image; bicubic(·) represents the bicubic interpolation processing; Feature extraction: Multispectral image M and F M ′ to stitch and extract features, and combine the full-color image P with F P ′After affine transformation, the features are extracted. The expressions are as follows: Where, To meet the requirements of time-frequency domain characteristics of multispectral images; To meet the requirements of the time-frequency domain characteristics of the full-color image.

6. The remote sensing image pan-sharpening method according to claim 1, characterized in that: The process of generating the modal fusion result by the modal fusion module includes: Extract QKV triples: Divide the spatial features or time-frequency domain features of the multispectral image and the panchromatic image into three parts respectively to serve as the query, key, and value of the attention mechanism. The expressions are as follows: Q M ,K M ,V M =Split(F M ) Q P ,K P ,V P =Split(F P ) Where, F M is the spatial feature or time-frequency domain feature of the multispectral image, F P is the spatial feature or time-frequency domain feature of the full-color image, Q M , K M and V M They are respectively from feature F M Query, Key and Value; Q P , K P and V P They are respectively from feature F P Query, Key and Value; Split(·) means equal split processing; Generate global token: F is trained through multi-layer perceptron and pooling layer M and F P The serialized result x M and x P Processing is performed to obtain the global token, the expression is as follows: token G =AvgPool(MLP([x M ,x P ])) Where token G represents the global token; AvgPool(·) represents average pooling; MLP(·) represents multi-layer perceptron processing; Extract difference features: Use the similarity matrix in the attention mechanism to extract F M and F P Common features, using V M After subtraction, it is used as the difference feature and adjusted using the linear layer. The expression is as follows: F D =Linear(V M -Attention(Q P ,K M ,V M )) Where, F D is the adjusted difference feature; Linear(·) is the linear layer processing; Attention(·) represents the similarity matrix processing in the attention mechanism calculation; Extract modal fusion results: Extract the common features twice and add them to the difference features to obtain the complete image features. Then use convolution to perform upsampling to obtain the modal fusion results. The expression is as follows: F C1 =Attention(F D ,K M ,V M ) F C2 =Attention(F C1 ,K P ,V P ) F O =Conv(F D +F C2 ) Where, F C1 and F C2 are the results of extracting common features twice, F O is the modal fusion result.

7. The remote sensing image pan-sharpening method according to claim 6, characterized in that: The process of generating a high-resolution spectral image by the reconstruction module includes: Serialization: Serialize the noisy image N into a token sequence x N ; Generate conditional injection parameters: By taking the modal fusion result F O The conditional injection parameters are evenly divided and expressed as follows: γ1,β1,α1,KV,α2,γ2,β2,α3=Split(F O ) In the formula, the reconstruction module is equipped with reorganization normalization; γ1~γ2 are used for scaling in reorganization normalization; β1~β2 are used for bias in reorganization normalization; α1~α3 are used for gating; KV represents the key and value features provided from the conditions in the cross attention; Image denoising: Use the self-attention mechanism to reconstruct the image of the token sequence. The expression is as follows: x N_1 =SelfAttention([LN(γ1⊙x N +β1),token t ,token G ]) x N_2 =α1⊙x N +x N Where x N_k is the k-th level processing result of the token sequence; LN(·) represents layer normalization processing; SelfAttention(·) represents the self-attention mechanism; token t is the time token corresponding to time step t, t∈[1,2,…,T], T is the total number of time steps; Use the cross attention mechanism to N_2 Strengthen the conditional information, the expression is as follows: K C ,V C =Split(Conv(KV)) x N_3 =CrossAttention(LN(x N_2 ),K C ,V C ) x N_4 =α2⊙x N_3 +x N_3 Where K C and V C are the Key and Value from the condition respectively; CrossAttention(·) is the cross attention mechanism; Use multi-layer perceptron to n_4 After adjustment, the expression is as follows: x N_5 =MLP(LN(γ2⊙x N_4 +β2)) x N_6 =α3⊙x N_5 +x N_5 Image reconstruction: Repeat the above steps several times and use the linear layer to reconstruct the final x N_6 Down-sample and then add it to the multispectral image M to generate a high-resolution multispectral image I H : I H =Rearrange(Linear(x N_6 ))+M Where Linear(·) represents linear layer processing; Rearrange(·) represents reorganization.

8. The remote sensing image pan-sharpening method according to claim 1, characterized in that: The process of generating time tokens by the time step embedding module includes: Generate time position encoding: generate a discrete time step range [1, 2, ..., T] based on the total number of time steps T; The time step t∈[1,2,…,T] is encoded using the sine and cosine functions, as shown below: Where TE(t) is the encoding of time step t; dim is the encoding dimension; m is the dimension index; % means remainder; Use a multi-layer perceptron to convert the encoding of time step t into a time token, as shown below: token t =MLP(TE(t)) Where token t is the time token corresponding to time step t; MLP(·) represents multi-layer perceptron processing; The process of forming a noisy image by the noise scheduling module includes: The accumulation degree of noise intensity is controlled by the cosine function, and the expression is as follows: Where, represents the cumulative retention factor from time step 0 to time step t; is a smoothing parameter; During the model training phase, noise is added to the original image to generate a noisy image. The noisy image expression at time t is as follows: Where x t represents the noisy image corresponding to time step t; x0 represents the original image; ∈ is Gaussian noise; When the actual model is sampled and predicted, the sub-time step sequence [τ1,τ2,…,τ s ], s is the total number of sub-time steps, so the accelerated sampling is performed according to the following formula: Where, x θ (·) represents the denoising network, τ represents the current sampling sub-time step, τ∈[τ1,τ2,…,τ s ], x τ represents the noisy image at the sampling stage sub-time step τ, c is the multispectral image and the panchromatic image, is the predicted value of x0 by the denoising network; is the noise intensity at sub-time step τ.

9. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the remote sensing image full-color sharpening method according to any one of claims 1 to 8 are implemented.

Images