Remote sensing image panchromatic sharpening method, model training method and system

Abstract

The invention relates to the field of image processing, in particular to a remote sensing image panchromatic sharpening method and a model training method and system. The training method comprises the steps that panchromatic-spectrum fusion features of a panchromatic image and a multispectral image and time information serve as prior information, and the prior information and a diffusion noise-added image are input into a diffusion model so that a fitting noise-added image can contain more panchromatic information and spectrum information; predicting the diffusion noise adding image and the panchromatic space-spectrum fusion feature through a Transform module so as to generate a reconstructed difference image in which complete panchromatic information and spectrum information are reserved; the method has remarkable advantages in the aspects of noise suppression and detail enhancement, and a multispectral image with more sufficient spectral information and spatial information can be generated.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a remote sensing image full color sharpening method, a model training method and a system. Background Art

[0002] Pan-sharpening of remote sensing images is an important task in the field of remote sensing image processing. It is mainly used to fuse low-resolution multispectral images with high-resolution panchromatic images to generate an image with high spatial resolution and rich spectral information. This is particularly important in the fields of remote sensing imaging and satellite image processing. However, traditional pan-sharpening methods have limitations in terms of spectral information fidelity, detail enhancement, and noise suppression, and are difficult to meet the needs of high-precision image processing.

[0003] In recent years, with the rapid development of deep learning technology and computer vision, deep neural networks have gradually replaced traditional methods and been applied in full-color sharpening tasks. Existing deep neural network full-color sharpening methods can be roughly divided into two categories: methods based on convolutional neural networks (CNNs) and methods based on transformers. Among them, the CNN-based methods can effectively extract local features of images. However, CNN always maintains the same weight matrix for different image regions, while the relationship between pixels in local regions in the full-color sharpening task is different, and the CNN method cannot achieve long-distance dependencies, which often affects the generation of the final predicted image. For methods using transformers, transformers use self-attention mechanisms to capture global interactions between contexts and have the ability to learn global information from images. However, transformers themselves divide the input image into fixed-size image blocks and process them independently, which makes it difficult for transformer-based methods to learn pixel-level attention and thus difficult to obtain local fine features of images.

[0004] In view of the advantages and disadvantages of CNN and Transformer in the task of full-color sharpening, technicians began to explore diffusion models, which are a type of generative model used to model and sample complex data distributions. They simulate the process of random walks and continuously update the sample distribution at each time step, thereby gradually approaching the target distribution. At present, most technicians in the full-color sharpening task use a method that combines the diffusion model with CNN. This method takes advantage of the advantages of CNN in extracting local features and the characteristics of the diffusion model in generating detailed information. It can generate high-resolution multispectral images with rich detailed information, but the method of combining the diffusion model with CNN is not perfect for the generation of spectral information. In the field of full-color sharpening, a small number of technicians have also proposed a method that combines the diffusion model with the Transformer. This method effectively generates high-resolution multispectral images with relatively richer spectral information, but there are still problems with blurred edges and spectral distortion. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a remote sensing image full-color sharpening method, a model training method and a system. On the basis of combining the diffusion model with the Transformer method, the full-color information, spectral information and time information are used as the prior information of the diffusion model. Through the frequency cross-attention mechanism, the Transformer module can better predict the distribution of noise in the diffusion model. The former focuses on the generation of stable images, so that the image can retain more complete spatial and spectral information; the latter assigns different attention weights to different frequency components to achieve the purpose of capturing key frequency information and avoid the problem of spectral distortion.

[0006] In a first aspect of the present invention, the present invention provides a remote sensing image pan-sharpening model training method, the method comprising:

[0007] Acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image;

[0008] Input the panchromatic image and the multispectral image into a feature extraction module, and output panchromatic image features and spectral image features;

[0009] Inputting the panchromatic image feature and the spectral image feature into a feature fusion module, and outputting a panchromatic-spectral fusion feature;

[0010] Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image;

[0011] Input the diffusion-noised image and the panchromatic-spectral fusion feature into a Transformer module, and output a fitted noisy image with panchromatic-spectral information;

[0012] Inputting the fitted noise image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image;

[0013] Adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0014] According to the difference image between the high-resolution multispectral image and the real image, the loss value of the remote sensing image pan-sharpening model is calculated; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model and a Transformer module;

[0015] According to the loss value, the model parameters of the remote sensing image pan-sharpening model are updated through gradients.

[0016] In a second aspect of the present invention, the present invention proposes a remote sensing image panchromatic sharpening method, the method comprising:

[0017] Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0018] Inputting the target panchromatic image, the target multispectral image and Gaussian noise into the remote sensing image panchromatic sharpening model trained as in the first aspect of the present invention, and outputting a reconstructed difference image of the target remote sensing image;

[0019] The reconstructed difference image is added to the target multispectral image to obtain a high-resolution target multispectral image.

[0020] In a third aspect of the present invention, the present invention further proposes a remote sensing image pan-sharpening model training system, the training system comprising:

[0021] An acquisition module is used to acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image and a real image;

[0022] A feature extraction module, used for inputting the panchromatic image and the multispectral image into the feature extraction module, and outputting panchromatic features and spectral features;

[0023] A feature fusion module, used for inputting the panchromatic feature and the spectral feature into the feature fusion module, and outputting a panchromatic-spectral fusion feature;

[0024] a diffusion model processing module, used for inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image; and used for inputting the fitted noise image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image;

[0025] A Transformer processing module, used for processing the diffusion noise image and the panchromatic-spectral fusion feature to generate a priori noise image;

[0026] An image processing module, used for adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0027] The model training module is used to calculate the loss value of the remote sensing image pan-sharpening model according to the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image pan-sharpening model according to the loss value; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

[0028] In a fourth aspect of the present invention, the present invention further proposes a remote sensing image full color sharpening system, wherein the sharpening system adopts the remote sensing image full color sharpening model trained by the training system according to the third aspect of the present invention, and comprises:

[0029] An input module is used to obtain a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0030] A first processing module, used for inputting the target panchromatic image and the target multispectral image into a trained remote sensing image panchromatic sharpening model, and outputting a reconstructed difference image of the target remote sensing image;

[0031] A second processing module, configured to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image;

[0032] The output module is used to display the high-resolution target multispectral image.

[0033] The beneficial effects of the present invention are:

[0034] The remote sensing image full-color sharpening method, model training method and system proposed in the present invention give full play to the advantages of the Transformer module and the diffusion model, use the advantages of the Transformer in extracting global features, effectively fit the noise with spatial-spectral information, improve the pertinence of the diffusion model sampling capability, and enable the generated image to retain more complete spatial and spectral information; thereby, high-resolution multispectral images can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a remote sensing image pan-sharpening model training method according to an embodiment of the present invention;

[0036] Figure 2 is a flow chart of panchromatic image feature and spectral image feature extraction according to an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of the structure of a feature extraction module according to an embodiment of the present invention;

[0038] Figure 4 is a flow chart of panchromatic image feature and spectral image feature extraction according to another embodiment of the present invention;

[0039] Figure 5 is a flow chart of fitting and noisy image generation according to an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of the structure of the Transformer module in an embodiment of the present invention;

[0041] Figure 7 is a flowchart of preliminary fusion feature extraction according to an embodiment of the present invention;

[0042] Figure 8 is a schematic diagram of the structure of a remote sensing image pan-sharpening model according to an embodiment of the present invention;

[0043] Fig. 9 is a flow chart of a remote sensing image pan-sharpening method according to an embodiment of the present invention;

[0044] Fig.10 Schematic diagram of the structure of a remote sensing image pan-sharpening model training system according to an embodiment of the present invention;

[0045] Fig.11 Schematic diagram of the remote sensing image pan-sharpening system architecture according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the following description, the terms "first\second\third\fourth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Figure 1is a flow chart of a remote sensing image pan-sharpening model training method according to an embodiment of the present invention; Figure 1 As shown, the execution subject of the method may be an electronic device, such as a server, etc. The method comprises:

[0049] 101. Acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image;

[0050] In the embodiment of the present invention, the panchromatic image, multispectral image and real image of the same photographed object are taken together as a group of training remote sensing images, so that their respective advantages can be fully utilized. The high spatial resolution of the panchromatic image is used to capture the surface details, and the spectral information of the multispectral image is used to identify different surface cover types. The real image is used as a label to guide the training process of the remote sensing image panchromatic sharpening model, thereby generating a high-resolution multispectral image.

[0051] Among them, in remote sensing image processing, panchromatic images are captured in a wider spectral range (such as the visible light range), so they contain rich spatial information. Panchromatic images are usually high-resolution images; multispectral images are images captured in multiple discrete spectral bands, each band corresponds to a different spectral range, and are used to provide information about the composition and state of surface materials. These multispectral images are usually low-resolution images; and real images are images that have been accurately measured or verified and are used as reference standards in the training or verification process. Real images can include high-resolution satellite images, aerial photography images, ground-based measured data, etc. Real images are used to evaluate the performance of the model during training to ensure that the model can accurately extract useful information from the input panchromatic and multispectral images.

[0052] It is understandable that each set of training remote sensing images can be obtained through existing public data sets such as the GF2 data set, or can be a full-color image taken by a remote sensing satellite, an unmanned aerial vehicle, or other high-altitude platform, a multispectral image taken by a multispectral camera or remote sensing equipment, or a real image taken by a remote sensing satellite, an unmanned aerial vehicle, other high-altitude platform, or a multispectral camera or remote sensing equipment.

[0053] In some embodiments, after acquiring the training remote sensing images, each group of training remote sensing images can also be preprocessed, for example, based on the mean and standard deviation of each training remote sensing image, the corresponding training remote sensing image to be processed is calculated; for each training remote sensing image to be processed of multiple groups of training remote sensing images, data enhancement is performed on each training remote sensing image to be processed; the data enhancement includes random flipping, random blurring and random color adjustment.

[0054] In particular, after acquiring multiple sets of training remote sensing images, the multispectral images of each set of training remote sensing images are upsampled before being fused with the panchromatic images; upsampling can ensure that the multispectral images are spatially consistent with the panchromatic images. This helps to generate fused images with higher spatial resolution and spectral information.

[0055] 102. Input the panchromatic image and the multispectral image into a feature extraction module, and output panchromatic image features and spectral image features;

[0056] In some embodiments, Figure 2 As shown, the panchromatic image and the multispectral image are input into a feature extraction module, and the output panchromatic image features and spectral image features include:

[0057] 201. In a first stage, the panchromatic image and the multispectral image are input into a first feature extraction unit with a spatial attention module, and a first-stage panchromatic image feature map and a first-stage multispectral image feature map are output;

[0058] 202. In the second stage, the first-stage panchromatic image feature map and the first-stage multispectral image feature map are input into a second feature extraction unit with a channel attention module, and the second-stage panchromatic image feature map and the second-stage multispectral image feature map are output;

[0059] The second-stage panchromatic image feature map is used to indicate panchromatic image features, and the second-stage multispectral image feature map is used to indicate spectral image features.

[0060] It can be understood that the first stage realizes the interaction of spatial attention between panchromatic image and multispectral image, and the second stage realizes the interaction of channel attention between panchromatic image and multispectral image. There are two branches in each stage, and the two branches are also processed interactively. This can promote the performance of the panchromatic sharpening model of remote sensing images and improve the quality and visual effect of multispectral images.

[0061] In some embodiments, Figure 3As shown, the feature extraction module includes two-stage feature extraction units; the first feature extraction unit includes a first average pooling layer, a second average pooling layer, a first maximum pooling layer, a second maximum pooling layer, a first multi-layer perceptron and a first spatial attention module; the second feature extraction unit includes a third average pooling layer, a third maximum pooling layer, a second multi-layer perceptron and a first channel attention module; the first feature extraction unit is mainly used to obtain the spatial information of the full-color image and the multi-spectral image; the second feature extraction unit is mainly used to obtain the channel information of the full-color image and the multi-spectral image after the spatial information is fused; through the interactive ability of the first feature extraction unit and the second feature extraction unit, the full-color image features and spectral image features extracted by the feature extraction module are more complete.

[0062] In some embodiments, Figure 4 As shown, the panchromatic image and the multispectral image are input into a feature extraction module, and outputting panchromatic image features and spectral image features further comprises:

[0063] 211. Input the panchromatic image pan to the first average pooling layer and the first maximum pooling layer respectively, input the multispectral image lms to the second average pooling layer and the second maximum pooling layer respectively, and output the maximum features and average features of the panchromatic image and the multispectral image;

[0064] 212. After interacting with each other through a first spatial attention module, the maximum features and the average features of the panchromatic image and the multispectral image are input into a first multilayer perceptron, and the spatial feature weights of the panchromatic image and the multispectral image are output;

[0065] 213. Multiply the spatial feature weight of the full-color image by the full-color image and perform residual connection to obtain a first-stage full-color image feature map pan';

[0066] 214. Multiply the spatial feature weight of the multispectral image by the multispectral image and perform residual connection to obtain a first-stage multispectral image feature map lms';

[0067] 215. After the first-stage panchromatic image feature map pan' and the first-stage multispectral image feature map lms' interact with each other through the first channel attention module, they are respectively input into the third maximum pooling layer and the third average pooling layer, and the maximum feature and the average feature after the first-stage panchromatic image feature map and the first-stage multispectral image feature map interact with each other are output;

[0068] 216. Input the maximum feature and the average feature after the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map into a second multi-layer perceptron, and output the channel feature weights of the first-stage panchromatic image feature map and the first-stage multispectral image feature map;

[0069] 217. Multiply the first-stage full-color image feature map by the channel feature weight of the first-stage full-color image feature map and perform residual connection to obtain the second-stage full-color image feature map pan”;

[0070] 218. Multiply the first-stage multispectral image by the channel feature weight of the first-stage multispectral image and perform residual connection to obtain a second-stage multispectral image feature map lms".

[0071] This embodiment realizes the interaction of spatial attention between full-color image and multi-spectral image through the first stage, and realizes the interaction of channel attention between full-color image and multi-spectral image in the second stage. Each stage has two interactive branches. In the first stage, each branch has a maximum pooling layer and an average pooling layer. The maximum pooling layer and the average pooling layer of the two branches are merged, and then the feature weights corresponding to each branch are obtained after being passed into the multi-layer perceptron. After that, the inputs of the two branches in the first stage are multiplied by their respective feature weights and residual connections are performed to obtain the two outputs of the first stage. In the second stage, the two outputs of the first stage are merged, and then passed through the maximum pooling layer and the average pooling layer, and then passed through the multi-layer perceptron layer to split into the feature weights of each branch, and then multiplied and added with the two branch inputs of the second stage to obtain the full-color image features and the spectral image features. This can further promote the performance of the panchromatic sharpening model of remote sensing images; improve the quality and visual effects of multi-spectral images.

[0072] 103. Input the panchromatic image feature and the spectral image feature into a feature fusion module, and output a panchromatic-spectral fusion feature;

[0073] In some embodiments, the feature fusion module may use a residual module to fuse the panchromatic features and the spectral features; it may also use fusion based on element-level operations (such as addition, multiplication, etc.), fusion based on feature map splicing, etc.

[0074] In some embodiments, the spatial features and spectral features are passed into a three-layer residual module, each residual module has two 3×3 convolutional layers and one ReLu layer, and finally a full-color-spectral fusion feature is obtained.

[0075] 104. Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image;

[0076] It should be noted that diffusion models are a type of deep learning method based on probabilistic generation models, which mainly include forward diffusion process and reverse generation process. In the forward diffusion process, noise is gradually added to the data to transform it into pure noise. This process is usually a Markov chain, with a small amount of Gaussian noise added at each step. Through this process, the data distribution is gradually transformed into a Gaussian noise distribution. In the reverse generation process, learning is done to gradually remove noise from the noise and restore the original data. This process is also a Markov chain, but in the opposite direction, with gradual denoising. Through this process, starting from standard Gaussian noise, through the learned reverse diffusion process, the noise is gradually removed to generate new data consistent with the real data distribution.

[0077] In some embodiments, during the forward diffusion process, this embodiment adds a certain amount of Gaussian noise ∈ to the difference image between the real image and the multispectral image at each time step t. t Through this process, the data distribution is gradually transformed into a Gaussian noise distribution, thereby obtaining a diffuse noise image, which is a difference image after adding Gaussian noise. The time step number t of the diffusion process is a discrete integer representing the number of times the noise is gradually added; the initialization of this parameter directly determines the length and refinement of the diffusion process. Among them, the Gaussian noise here is a normally distributed random noise, and the embodiment of the present invention does not limit the specific Gaussian noise value.

[0078] 105. Input the diffusion-noised image and the panchromatic-spectral fusion feature into a Transformer module, and output a fitted-noised image with panchromatic-spectral information;

[0079] In some embodiments, Figure 5 As shown, the fitted noise-added image is output through the following process:

[0080] 501. Using the panchromatic-spectral fusion feature as prior information; using the diffusion-noised image as an input image;

[0081] 502. Input the input image and the prior information into a pre-fusion module of each Transformer layer in the Transformer module, and output a preliminary fusion feature;

[0082] 503. Extract the key vector K and the value vector V of the prior information, and extract the query vector Q of the preliminary fusion feature;

[0083] 504. Perform Fourier transform on the query vector Q, the key vector K, and the value vector V, multiply the query vector Q and the key vector K by a cross attention mechanism in the frequency domain, perform a softmax operation, and then multiply the query vector Q and the value vector V and perform an inverse Fourier transform to obtain a hierarchical fusion feature of the diffused noisy image and the panchromatic-spectral fusion feature;

[0084] 505. Input the hierarchical fusion features to the third multi-layer perceptron layer of the corresponding level, and use the output result of the third multi-layer perceptron of the corresponding level as the input data of the next Transformer layer in the Transformer module, until the third multi-layer perceptron of the last level outputs the prior noisy image.

[0085] This embodiment realizes cross-attention in the frequency domain and can assign different attention weights to different frequency components to achieve the purpose of capturing key frequency information, avoiding the problem of spectral distortion, and enabling the prior noisy image to retain more complete full-color information and spectral information.

[0086] In some embodiments, Figure 6 As shown, the Transformer module is composed of a Patch Embedding, twelve Transformer layers, an LN layer and an MLP layer in series. Patch Embedding divides the input into multiple image blocks and adds position encoding. Each Transformer layer includes a pre-fusion module and a Transformer encoder, wherein the pre-fusion module includes a second channel attention module and a 3×3 convolutional layer, the second channel attention module includes an average pooling layer, a maximum pooling layer and a fully connected layer, and the channel attention operation is implemented in the frequency domain, wherein the fully connected layer is implemented by a multi-layer perceptron MLP layer.

[0087] In some embodiments, Figure 7 As shown, the input image and the prior information are input into the pre-fusion module of each Transformer layer in the Transformer module, and the output of the preliminary fusion feature further includes:

[0088] 511. Input the prior information into a second channel attention module, and output a channel attention feature; the second channel attention module includes a fourth average pooling layer and a fourth maximum pooling layer; the channel attention feature includes a fused average pooling feature and a maximum pooling feature;

[0089] 512. Input the channel attention feature into a fourth multi-layer perceptron and multiply it by the prior information, and then output the prior channel feature;

[0090] 513. Input the fitted noise image into a convolutional layer, and output a fitted noise feature;

[0091] 514. Concatenate the priori channel feature and the fitted noise feature to obtain a preliminary fusion feature.

[0092] This embodiment allocates different attention weights to different frequency components through the second channel attention module to achieve the purpose of capturing key frequency information and avoid the problem of spectral distortion.

[0093] 106. Input the fitted noise image into a diffusion model, and output a reconstructed difference image between the real image and the multispectral image;

[0094] Corresponding to step 104, in some embodiments, during the inverse generation process, this embodiment removes a certain amount of Gaussian noise from the fitted noise image at each time step t. Through this process, starting from standard Gaussian noise, through the learned inverse diffusion process, the noise is gradually removed to generate a reconstructed difference image with the same distribution as the original difference image, and the reconstructed difference image is the difference image after removing the Gaussian noise.

[0095] 107. Add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0096] In this embodiment, the reconstructed difference image obtained in step 106 is superimposed on the multispectral image acquired in step 101 , so that a high-resolution multispectral image can be obtained.

[0097] 108. Calculate the loss value of the remote sensing image pan-sharpening model according to the difference image between the high-resolution multispectral image and the real image;

[0098] See also Figure 8 ,like Figure 8 As shown, the remote sensing image full-color sharpening model includes a feature extraction module, a feature fusion module, a diffusion model and a Transformer module, wherein the diffusion model is embodied by superimposing Gaussian noise, which is not shown; on the one hand, the full-color image PAN and the upsampled multispectral image LMS↑4 are input into the feature extraction module, and after passing through the feature fusion module, they are input into the Transformer model together with the time information T as prior information; on the other hand, the difference image between the real image GT and the upsampled multispectral image LMS↑4 is added with noise and input into the Transformer model together; the Transformer model generates a reconstructed difference image between the real image GT and the upsampled multispectral image LMS↑4; and then the reconstructed difference image is superimposed with the upsampled multispectral image LMS↑4 to obtain the final high-resolution multispectral image. Generate high-resolution multispectral images Corresponding to the real input image.

[0099] 109. According to the loss value, update the model parameters of the remote sensing image pan-sharpening model through gradient.

[0100] In one possible implementation, the method for obtaining the panchromatic sharpening model loss of the remote sensing image based on the difference image between the high-resolution multispectral image and the real image can specifically be to perform cross entropy loss calculation on the difference image between the high-resolution multispectral image and the real image to obtain the model loss.

[0101] It should be understood that the above-mentioned process of obtaining the model loss is only illustrative. If there are other intermediate features in the model training process (such as intermediate encoding features of one or more encoding layers in the encoding process or features in the decoding process), the model loss can also be obtained based on the intermediate features and the above-mentioned features, or based on the intermediate features and the above-mentioned encoding features, labels and image processing results. The model loss is obtained, which is not described one by one in this embodiment.

[0102] When adjusting the model parameters of the remote sensing image pan-sharpening model based on the model loss, the purpose is to minimize the model loss so that the remote sensing image pan-sharpening model gradually converges. When the number of times the remote sensing image pan-sharpening model parameters are adjusted reaches a preset number, for example, 1000 times, 10000 times, etc., the remote sensing image pan-sharpening model can be considered to have converged. It can also be when the model loss gradually tends to a fixed value, such as zero, or is less than a preset value, such as 0.05, 0.01, etc., then the model can be considered to have converged, and the trained remote sensing image pan-sharpening model can be obtained.

[0103] See also Fig. 9 The present application also proposes a remote sensing image full color sharpening method, the method comprising:

[0104] 111. Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0105] 112. Input the target panchromatic image, the target multispectral image and Gaussian noise into a trained remote sensing image panchromatic sharpening model, and output a reconstructed difference image of the target remote sensing image;

[0106] 113. Add the reconstructed difference image to the target multispectral image to obtain a high-resolution target multispectral image.

[0107] By adopting the above-mentioned remote sensing image panchromatic sharpening method of the present application, since the model used in the remote sensing image panchromatic sharpening process is a model trained using the aforementioned remote sensing image panchromatic sharpening model training method, therefore, when the above-mentioned remote sensing image panchromatic sharpening model is used during the sharpening process, the accuracy and precision of the extracted features can be improved. Therefore, when image sharpening is performed based on the extracted features, the generation accuracy of high-resolution multispectral images can be effectively improved.

[0108] See also Fig.10 , an embodiment of the present application provides a remote sensing image full color sharpening model training system:

[0109] An acquisition module 301 is used to acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image and a real image;

[0110] A feature extraction module 302, used for inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic features and spectral features;

[0111] A feature fusion module 303 is used to input the panchromatic feature and the spectral feature into a feature fusion module, and output a panchromatic-spectral fusion feature;

[0112] The diffusion model processing module 304 is used to input the difference image between the real image and the multispectral image and Gaussian noise into the diffusion model, and output a diffusion noise image; and to input the fitted noise image into the diffusion model, and output a reconstructed difference image between the real image and the multispectral image;

[0113] Transformer processing module 305, used for processing the diffusion noise image and the panchromatic-spectral fusion feature to generate a priori noise image;

[0114] An image processing module 306 is used to add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0115] The model training module 307 is used to calculate the loss value of the remote sensing image pan-sharpening model according to the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image pan-sharpening model according to the loss value; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

[0116] See also Fig.11 , an embodiment of the present application provides a remote sensing image full color sharpening system:

[0117] An input module 401 is used to obtain a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0118] A first processing module 402 is used to input the target panchromatic image and the target multispectral image into a trained remote sensing image pan-sharpening model, and output a reconstructed difference image of the target remote sensing image;

[0119] A second processing module 403 is used to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image;

[0120] The output module 404 is used to display the high-resolution target multispectral image.

[0121] The input module 401 may be a remote sensing satellite, a drone or other high-altitude platform, a multispectral camera or remote sensing equipment, etc., which can collect high-resolution full-color images of targets and low-resolution multispectral images of targets; the output module 404 may be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode), etc., and can display high-resolution multispectral images of targets.

[0122] Each module in the above-mentioned remote sensing image full-color sharpening model training system and remote sensing image full-color sharpening system can be fully or partially implemented by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the device embodiment in this application corresponds to the aforementioned method embodiment. The specific principles in the system embodiment can be found in the contents of the aforementioned method embodiment, which will not be repeated here.

[0123] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image pan-sharpening model training method, characterized in that: The method comprises: Acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image; Input the panchromatic image and the multispectral image into a feature extraction module, and output panchromatic image features and spectral image features; Inputting the panchromatic image feature and the spectral image feature into a feature fusion module, and outputting a panchromatic-spectral fusion feature; Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image; Input the diffusion-noised image and the panchromatic-spectral fusion feature into a Transformer module, and output a fitted noisy image with panchromatic-spectral information; Inputting the fitted noise image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image; Adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image; According to the difference image between the high-resolution multispectral image and the real image, the loss value of the remote sensing image pan-sharpening model is calculated; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model and a Transformer module; According to the loss value, the model parameters of the remote sensing image pan-sharpening model are updated through gradients.

2. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: After obtaining the multiple sets of training remote sensing images, the method further includes performing data preprocessing on the multiple sets of training remote sensing images; the data preprocessing includes calculating the training remote sensing images to be processed corresponding to the corresponding training remote sensing images according to the mean and standard deviation of each training remote sensing image; For each to-be-processed training remote sensing image of the multiple groups of training remote sensing images, data enhancement is performed on each to-be-processed training remote sensing image; the data enhancement includes random flipping, random blurring and random color adjustment.

3. The remote sensing image pan-sharpening model training method according to claim 1 or 2, characterized in that: After acquiring a plurality of sets of training remote sensing images, the method further includes up-sampling the multispectral images of each set of training remote sensing images.

4. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features include: In the first stage, the panchromatic image and the multispectral image are input into a first feature extraction unit with a spatial attention module, and a first-stage panchromatic image feature map and a first-stage multispectral image feature map are output; the first feature extraction unit includes a first average pooling layer, a second average pooling layer, a first maximum pooling layer, a second maximum pooling layer, a first multilayer perceptron and a first spatial attention module; In the second stage, the panchromatic image feature map of the first stage and the multispectral image feature map of the first stage are input into a second feature extraction unit with a channel attention module, and the second stage panchromatic image feature map and the second stage multispectral image feature map are output; the second feature extraction unit includes a third average pooling layer, a third maximum pooling layer, a second multilayer perceptron and a first channel attention module; The second-stage panchromatic image feature map is used to indicate panchromatic image features, and the second-stage multispectral image feature map is used to indicate spectral image features.

5. The remote sensing image pan-sharpening model training method according to claim 4, characterized in that: Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features specifically includes: Input the panchromatic image to a first average pooling layer and a first maximum pooling layer, respectively, input the multispectral image to a second average pooling layer and a second maximum pooling layer, respectively, and output the maximum features and average features of the panchromatic image and the multispectral image; After interacting with the maximum features and average features of the panchromatic image and the multispectral image through a first spatial attention module, the maximum features and the average features are input into a first multi-layer perceptron, and the spatial feature weights of the panchromatic image and the multispectral image are output; Multiplying the spatial feature weight of the full-color image by the full-color image and performing residual connection to obtain a first-stage full-color image feature map; The spatial feature weight of the multispectral image is multiplied by the multispectral image and then residual connection is performed to obtain a first-stage multispectral image feature map; After the first-stage panchromatic image feature map and the first-stage multispectral image feature map interact with each other through the first channel attention module, they are respectively input into the third maximum pooling layer and the third average pooling layer, and the maximum feature and the average feature after the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map are output; Inputting the maximum feature and the average feature after the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map into the second multi-layer perceptron, and outputting the channel feature weights of the first-stage panchromatic image feature map and the first-stage multispectral image feature map; The first-stage full-color image feature map is multiplied by the channel feature weight of the first-stage full-color image feature map, and then a residual connection is performed to obtain a second-stage full-color image feature map; The first-stage multispectral image is multiplied by the channel feature weight of the first-stage multispectral image and then residual connection is performed to obtain a second-stage multispectral image feature map.

6. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: The step of inputting the diffusion noise image and the panchromatic-spectral fusion feature into a Transformer module and outputting a priori noise image comprises: Using the panchromatic-spectral fusion feature as prior information; using the diffusion-noised image as an input image; Inputting the input image and the prior information into a pre-fusion module of each Transformer layer in the Transformer module, and outputting preliminary fusion features; Extracting the key vector K and the value vector V of the prior information, and extracting the query vector Q of the preliminary fusion feature; The query vector Q, the key vector K, and the value vector V are Fourier transformed, and the query vector Q is multiplied by the key vector K in the frequency domain through the cross attention mechanism, and then a softmax operation is performed, and then the query vector Q is multiplied by the value vector V and then an inverse Fourier transform is performed to obtain the hierarchical fusion feature of the diffusion noise image and the panchromatic-spectral fusion feature; The hierarchical fusion features are input into the third multi-layer perceptron layer of the corresponding level, and the output result of the third multi-layer perceptron of the corresponding level is used as the input data of the next Transformer layer in the Transformer module, until the third multi-layer perceptron of the last level outputs the prior noisy image.

7. The remote sensing image pan-sharpening model training method according to claim 6, characterized in that: The input image and the prior information are input into the pre-fusion module of each Transformer layer in the Transformer module, and the output preliminary fusion features include: The prior information is input into a second channel attention module, and a channel attention feature is output; the second channel attention module includes a fourth average pooling layer and a fourth maximum pooling layer; the channel attention feature includes a fused average pooling feature and a maximum pooling feature; Inputting the channel attention feature into a fourth multi-layer perceptron and multiplying it by the prior information, outputting the prior channel feature; Inputting the fitted noise image into a convolutional layer and outputting fitted noise features; The prior channel feature is concatenated with the fitted noise feature to obtain a preliminary fusion feature.

8. A remote sensing image panchromatic sharpening method, characterized in that: The method comprises: Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image; Inputting the target panchromatic image, the target multispectral image and Gaussian noise into any trained remote sensing image panchromatic sharpening model as described in claims 1 to 7, and outputting a reconstructed difference image of the target remote sensing image; The reconstructed difference image is added to the target multispectral image to obtain a high-resolution target multispectral image.

9. A remote sensing image full color sharpening model training system, characterized in that: The method is based on a remote sensing image full color sharpening model training method as claimed in any one of claims 1 to 7, wherein the training system comprises: An acquisition module is used to acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image and a real image; A feature extraction module, used for inputting the panchromatic image and the multispectral image into the feature extraction module, and outputting panchromatic features and spectral features; A feature fusion module, used for inputting the panchromatic feature and the spectral feature into the feature fusion module, and outputting a panchromatic-spectral fusion feature; a diffusion model processing module, used for inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image; and used for inputting the fitted noise image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image; A Transformer processing module, used for processing the diffusion noise image and the panchromatic-spectral fusion feature to generate a priori noise image; An image processing module, used for adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image; The model training module is used to calculate the loss value of the remote sensing image pan-sharpening model according to the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image pan-sharpening model according to the loss value; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

10. A remote sensing image full color sharpening system, characterized in that: The sharpening system adopts the remote sensing image full color sharpening model trained by the training system as claimed in claim 9, and comprises: An input module is used to obtain a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image; A first processing module, used for inputting the target panchromatic image and the target multispectral image into a trained remote sensing image panchromatic sharpening model, and outputting a reconstructed difference image of the target remote sensing image; A second processing module, configured to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image; The output module is used to display the high-resolution target multispectral image.

Images