Remote sensing image pan-sharpening method, model training method and system

Abstract

The present invention relates to the field of image processing, and in particular to a remote sensing image panchromatic sharpening method, model training method, and system. The training method comprises taking the panchromatic-spectral fusion features of a panchromatic image and a multispectral image, as well as time information, as prior information, and inputting the prior information and a diffused-noised image into a diffusion model so that the fitted noisy image can contain more panchromatic and spectral information. The diffused-noised image and the panchromatic spatial-spectral fusion features are predicted using a Transformer module to generate a reconstructed difference image that retains complete panchromatic and spectral information. The present invention has significant advantages in noise suppression and detail enhancement, and can generate a multispectral image with more sufficient spectral and spatial information.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a remote sensing image full color sharpening method, a model training method and a system. Background Art

[0002] Pan-sharpening of remote sensing images is a crucial task in remote sensing image processing. It fuses low-resolution multispectral images with high-resolution panchromatic images to produce an image with high spatial resolution and rich spectral information. This is particularly important in remote sensing imaging and satellite image processing. However, traditional pan-sharpening methods have limitations in terms of spectral fidelity, detail enhancement, and noise suppression, making them difficult to meet the demands of high-precision image processing.

[0003] In recent years, with the rapid development of deep learning technology and computer vision, deep neural networks have gradually replaced traditional methods in the application of pan-sharpening tasks. Existing deep neural network pan-sharpening methods can be roughly divided into two categories: convolutional neural network (CNN)-based methods and Transformer-based methods. CNN-based methods are effective in extracting local features of images. However, CNNs maintain the same weight matrix for different image regions, while the relationships between pixels in local regions in pan-sharpening tasks vary. Furthermore, CNNs cannot capture long- and short-range dependencies, which often affects the generation of the final predicted image. Transformer-based methods utilize a self-attention mechanism to capture global interactions between contexts, enabling them to learn global information from images. However, the Transformer itself divides the input image into fixed-sized patches and processes them independently, making it difficult for Transformer-based methods to learn pixel-level attention and, consequently, to capture fine local features of the image.

[0004] In view of the advantages and disadvantages of CNN and Transformer in full-color sharpening tasks, technicians began to explore diffusion models, which are a type of generative model used to model and sample complex data distributions. They simulate the process of random walks and continuously update the sample distribution at each time step, thereby gradually approaching the target distribution. At present, most technicians in full-color sharpening tasks use a method that combines the diffusion model with CNN. This method takes advantage of the advantages of CNN in extracting local features and the characteristics of the diffusion model in generating detailed information. It can generate high-resolution multispectral images with rich detailed information, but the method of combining the diffusion model with CNN is not perfect for the generation of spectral information. In the field of full-color sharpening, a small number of technicians have also proposed a method that combines the diffusion model with the Transformer. This method effectively generates high-resolution multispectral images with relatively richer spectral information, but there are still problems with edge blur and spectral distortion. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides a remote sensing image full-color sharpening method, model training method and system. Based on the method combining the diffusion model and the Transformer, the full-color information, spectral information and time information are used as the prior information of the diffusion model. Through the frequency cross-attention mechanism, the Transformer module can better predict the distribution of noise in the diffusion model. The former focuses on stabilizing the generation of the image so that the image can retain more complete spatial and spectral information; the latter assigns different attention weights to different frequency components to achieve the purpose of capturing key frequency information and avoid the problem of spectral distortion.

[0006] In a first aspect of the present invention, the present invention provides a remote sensing image pan-sharpening model training method, the method comprising:

[0007] Acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image;

[0008] Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features;

[0009] Inputting the panchromatic image features and the spectral image features into a feature fusion module, and outputting panchromatic-spectral fusion features;

[0010] Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image;

[0011] Input the diffusion-noised image and the panchromatic-spectral fusion features into a Transformer module, and output a fitted noisy image with panchromatic-spectral information;

[0012] Inputting the fitted noise image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image;

[0013] Adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0014] Calculating the loss value of a remote sensing image pan-sharpening model based on a difference image between the high-resolution multispectral image and the real image; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module;

[0015] According to the loss value, the model parameters of the remote sensing image pan-sharpening model are updated through gradients.

[0016] In a second aspect of the present invention, the present invention provides a remote sensing image pan-sharpening method, the method comprising:

[0017] Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0018] Inputting the target panchromatic image, the target multispectral image and Gaussian noise into the remote sensing image pan-sharpening model trained as in the first aspect of the present invention, and outputting a reconstructed difference image of the target remote sensing image;

[0019] The reconstructed difference image is added to the target multispectral image to obtain a high-resolution target multispectral image.

[0020] In a third aspect of the present invention, the present invention further proposes a remote sensing image pan-sharpening model training system, the training system comprising:

[0021] An acquisition module is used to acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image;

[0022] A feature extraction module, configured to input the panchromatic image and the multispectral image into the feature extraction module and output panchromatic features and spectral features;

[0023] A feature fusion module, configured to input the panchromatic feature and the spectral feature into a feature fusion module and output a panchromatic-spectral fusion feature;

[0024] a diffusion model processing module, configured to input a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model and output a diffusion noise image; and to input the fitted noise image into the diffusion model and output a reconstructed difference image between the real image and the multispectral image;

[0025] A Transformer processing module, configured to process the diffusion-noised image and the panchromatic-spectral fusion feature to generate a priori-noised image;

[0026] An image processing module, configured to add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0027] The model training module is used to calculate the loss value of the remote sensing image full-color sharpening model based on the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image full-color sharpening model based on the loss value; the remote sensing image full-color sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

[0028] In a fourth aspect, the present invention further provides a remote sensing image pan-sharpening system, wherein the pan-sharpening system uses the remote sensing image pan-sharpening model trained by the training system according to the third aspect of the present invention, and comprises:

[0029] An input module is used to obtain a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0030] A first processing module is configured to input the target panchromatic image and the target multispectral image into a trained remote sensing image pan-sharpening model, and output a reconstructed difference image of the target remote sensing image;

[0031] a second processing module, configured to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image;

[0032] The output module is used to display the high-resolution target multispectral image.

[0033] The beneficial effects of the present invention are:

[0034] The remote sensing image full-color sharpening method, model training method and system proposed in the present invention give full play to the advantages of the Transformer module and diffusion model, and use the advantages of the Transformer in extracting global features to effectively fit the noise with spatial-spectral information, thereby improving the targeted sampling capability of the diffusion model and enabling the generated image to retain more complete spatial and spectral information; thus, it is possible to generate high-resolution multispectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a remote sensing image pan-sharpening model training method according to an embodiment of the present invention;

[0036] Figure 2 is a flow chart of panchromatic image feature and spectral image feature extraction according to an embodiment of the present invention;

[0037] Figure 3 2 is a schematic diagram of the structure of a feature extraction module according to an embodiment of the present invention;

[0038] Figure 4 is a flow chart of panchromatic image feature and spectral image feature extraction according to another embodiment of the present invention;

[0039] Figure 5 is a flow chart of fitting and noisy image generation according to an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of the Transformer module structure of an embodiment of the present invention;

[0041] Figure 7 is a flowchart of preliminary fusion feature extraction according to an embodiment of the present invention;

[0042] Figure 8 1 is a schematic diagram of the structure of a remote sensing image pan-sharpening model according to an embodiment of the present invention;

[0043] Figure 9 is a flow chart of a remote sensing image pan-sharpening method according to an embodiment of the present invention;

[0044] Figure 10 1 is a schematic diagram of the structure of a remote sensing image pan-sharpening model training system according to an embodiment of the present invention;

[0045] Figure 11 2 is a schematic diagram of the architecture of a remote sensing image pan-sharpening system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] In the following description, the terms "first\second\third\fourth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Figure 1: is a flow chart of a remote sensing image pan-sharpening model training method according to an embodiment of the present invention; Figure 1 As shown, the execution subject of the method may be an electronic device, such as a server, etc. The method includes:

[0049] 101. Acquire multiple sets of training remote sensing images, each set of training remote sensing images including a panchromatic image, a multispectral image, and a real image;

[0050] In this embodiment of the present invention, panchromatic, multispectral, and real images of the same subject are combined as a training set of remote sensing images, leveraging their respective strengths. The high spatial resolution of the panchromatic image is used to capture surface details, while the spectral information of the multispectral image is used to identify different land cover types. The real images serve as labels to guide the training process of the remote sensing image pan-sharpening model, thereby generating high-resolution multispectral images.

[0051] In remote sensing image processing, panchromatic images are captured over a wide spectral range (such as the visible light range) and therefore contain rich spatial information. Panchromatic images are usually high-resolution images. Multispectral images are captured over multiple discrete spectral bands, each corresponding to a different spectral range, and are used to provide information about the composition and state of surface materials. These multispectral images are usually low-resolution images. Real images are precisely measured or verified images and are used as reference standards during training or verification. Real images can include high-resolution satellite images, aerial photography images, ground-truth data, and so on. Real images are used to evaluate the performance of the model during training to ensure that the model can accurately extract useful information from the input panchromatic and multispectral images.

[0052] It is understandable that each set of training remote sensing images can be obtained through existing public datasets such as the GF2 dataset, or can be full-color images taken by remote sensing satellites, drones or other high-altitude platforms, multispectral images taken by multispectral cameras or remote sensing equipment, or real images taken by remote sensing satellites, drones, other high-altitude platforms or multispectral cameras or remote sensing equipment.

[0053] In some embodiments, after obtaining the training remote sensing images, each group of training remote sensing images can also be preprocessed. For example, based on the mean and standard deviation of each training remote sensing image, the corresponding training remote sensing image to be processed is calculated; for each training remote sensing image to be processed in multiple groups of training remote sensing images, data enhancement is performed on each training remote sensing image to be processed; the data enhancement includes random flipping, random blurring and random color adjustment.

[0054] Specifically, after acquiring multiple sets of training remote sensing images, each set of multispectral images is upsampled before being fused with the panchromatic image. Upsampling ensures that the multispectral image is spatially consistent with the panchromatic image, which helps generate fused images with higher spatial resolution and spectral information.

[0055] 102. Input the panchromatic image and the multispectral image into a feature extraction module, and output panchromatic image features and spectral image features;

[0056] In some embodiments, as Figure 2 As shown, the panchromatic image and the multispectral image are input into a feature extraction module, and the output panchromatic image features and spectral image features include:

[0057] 201. In a first stage, input the panchromatic image and the multispectral image into a first feature extraction unit with a spatial attention module, and output a first-stage panchromatic image feature map and a first-stage multispectral image feature map;

[0058] 202. In the second stage, input the first-stage panchromatic image feature map and the first-stage multispectral image feature map into a second feature extraction unit with a channel attention module, and output the second-stage panchromatic image feature map and the second-stage multispectral image feature map;

[0059] The second-stage panchromatic image feature map is used to indicate panchromatic image features, and the second-stage multispectral image feature map is used to indicate spectral image features.

[0060] It can be understood that the first stage realizes the interaction of spatial attention between panchromatic images and multispectral images, and the second stage realizes the interaction of channel attention between panchromatic images and multispectral images. There are two branches in each stage, and the two branches are also interactively processed. This can promote the performance of the remote sensing image panchromatic sharpening model and improve the quality and visual effects of multispectral images.

[0061] In some embodiments, as Figure 3As shown, the feature extraction module includes two-stage feature extraction units; the first feature extraction unit includes a first average pooling layer, a second average pooling layer, a first maximum pooling layer, a second maximum pooling layer, a first multi-layer perceptron and a first spatial attention module; the second feature extraction unit includes a third average pooling layer, a third maximum pooling layer, a second multi-layer perceptron and a first channel attention module; the first feature extraction unit is mainly used to obtain the spatial information of the panchromatic image and the multispectral image; the second feature extraction unit is mainly used to obtain the channel information of the panchromatic image and the multispectral image after the spatial information is fused; through the interactive ability of the first feature extraction unit and the second feature extraction unit, the panchromatic image features and spectral image features extracted by the feature extraction module are more complete.

[0062] In some embodiments, as Figure 4 As shown, the panchromatic image and the multispectral image are input into a feature extraction module, and outputting panchromatic image features and spectral image features further includes:

[0063] 211. Input the panchromatic image pan into the first average pooling layer and the first maximum pooling layer, respectively, and input the multispectral image lms into the second average pooling layer and the second maximum pooling layer, respectively, and output the maximum features and average features of the panchromatic image and the multispectral image;

[0064] 212. After interacting with each other through a first spatial attention module, the maximum features and the average features of the panchromatic image and the multispectral image are input into a first multilayer perceptron, and the spatial feature weights of the panchromatic image and the multispectral image are output.

[0065] 213. Multiply the spatial feature weight of the panchromatic image by the panchromatic image and perform residual connection to obtain a first-stage panchromatic image feature map pan';

[0066] 214. Multiply the spatial feature weight of the multispectral image by the multispectral image and perform residual connection to obtain a first-stage multispectral image feature map lms';

[0067] 215. After the first-stage panchromatic image feature map pan' and the first-stage multispectral image feature map lms' are interacted through the first channel attention module, the interaction is respectively input into the third maximum pooling layer and the third average pooling layer, and the maximum feature and the average feature of the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map are output;

[0068] 216. Input the maximum feature and the average feature after the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map into a second multi-layer perceptron, and output the channel feature weights of the first-stage panchromatic image feature map and the first-stage multispectral image feature map;

[0069] 217. Multiply the first-stage full-color image feature map by the channel feature weight of the first-stage full-color image feature map and perform a residual connection to obtain a second-stage full-color image feature map “pan”;

[0070] 218. Multiply the first-stage multispectral image by the channel feature weights of the first-stage multispectral image and perform residual connection to obtain a second-stage multispectral image feature map lms".

[0071] This embodiment realizes the interaction of spatial attention between panchromatic image and multispectral image in the first stage, and realizes the interaction of channel attention between panchromatic image and multispectral image in the second stage. Each stage has two interactive branches. In the first stage, each branch has a maximum pooling layer and an average pooling layer. The maximum pooling layer and the average pooling layer of the two branches are merged, and then passed into the multi-layer perceptron and then split to obtain the feature weights corresponding to each branch. After that, the inputs of the two branches in the first stage are multiplied by their respective feature weights and residual connections are performed to obtain the two outputs of the first stage. In the second stage, the two outputs of the first stage are merged, and then passed through the maximum pooling layer and the average pooling layer, and then passed through the multi-layer perceptron layer to split into the feature weights of each branch, and then multiplied and added with the two branch inputs of the second stage to obtain the panchromatic image features and the spectral image features. This can further promote the performance of the remote sensing image panchromatic sharpening model and improve the quality and visual effects of multispectral images.

[0072] 103. Input the panchromatic image feature and the spectral image feature into a feature fusion module, and output a panchromatic-spectral fusion feature;

[0073] In some embodiments, the feature fusion module may use a residual module to fuse the panchromatic features and the spectral features; it may also use fusion based on element-level operations (such as addition, multiplication, etc.), fusion based on feature map splicing, etc.

[0074] In some embodiments, the spatial features and spectral features are passed into a three-layer residual module, each residual module has two 3×3 convolutional layers and one ReLu layer, and finally the full color-spectral fusion feature is obtained.

[0075] 104. Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image;

[0076] It should be noted that diffusion models are a type of deep learning method based on probabilistic generative models. They primarily involve a forward diffusion process and a reverse generation process. In the forward diffusion process, noise is gradually added to the data, transforming it into pure noise. This process is typically a Markov chain, with a small amount of Gaussian noise added at each step. Through this process, the data distribution is gradually transformed into a Gaussian noise distribution. In the reverse generation process, learning gradually removes the noise from the noise, restoring the original data. This process is also a Markov chain, but in the opposite direction, gradually removing noise. Starting with standard Gaussian noise, this process uses a learned reverse diffusion process to gradually remove the noise, generating new data that is consistent with the true data distribution.

[0077] In some embodiments, during the forward diffusion process, this embodiment adds a certain amount of Gaussian noise ∈ to the difference image between the real image and the multispectral image at each time step t. t Through this process, the data distribution is gradually transformed into a Gaussian noise distribution, resulting in a diffuse noise image. This diffuse noise image is the difference image after adding Gaussian noise. The time step number t of the diffusion process is a discrete integer representing the number of times noise is gradually added; the initialization of this parameter directly determines the length and refinement of the diffusion process. The Gaussian noise here is a normally distributed random noise, and the specific Gaussian noise value is not limited in this embodiment of the present invention.

[0078] 105. Input the diffusion-noised image and the panchromatic-spectral fusion feature into a Transformer module, and output a fitted noisy image with panchromatic-spectral information;

[0079] In some embodiments, as Figure 5 As shown, the fitted noise-added image is output through the following process:

[0080] 501. Using the panchromatic-spectral fusion feature as prior information; using the diffusion-noised image as an input image;

[0081] 502. Input the input image and the prior information into a pre-fusion module of each Transformer layer in the Transformer module, and output preliminary fusion features;

[0082] 503. Extract the key vector K and the value vector V of the prior information, and extract the query vector Q of the preliminary fusion feature;

[0083] 504. Perform Fourier transform on the query vector Q, the key vector K, and the value vector V. In the frequency domain, the query vector Q is multiplied by the key vector K using a cross-attention mechanism, followed by a softmax operation. The query vector Q is then multiplied by the value vector V and then inverse Fourier transform is performed to obtain a hierarchical fusion feature of the diffused noisy image and the panchromatic-spectral fusion feature.

[0084] 505. Input the hierarchical fusion features into the third multi-layer perceptron layer of the corresponding level, and use the output result of the third multi-layer perceptron of the corresponding level as the input data of the next Transformer layer in the Transformer module, until the third multi-layer perceptron of the last level outputs the prior noisy image.

[0085] This embodiment implements cross-attention in the frequency domain and can assign different attention weights to different frequency components to capture key frequency information, thereby avoiding the problem of spectral distortion and enabling the prior noisy image to retain more complete full-color and spectral information.

[0086] In some embodiments, as Figure 6 As shown in the figure, the Transformer module is composed of a Patch Embedding, twelve Transformer layers, an LN layer and an MLP layer in series. Patch Embedding divides the input into multiple image blocks and adds position encoding. Each Transformer layer includes a pre-fusion module and a Transformer encoder, where the pre-fusion module includes a second channel attention module and a 3×3 convolutional layer. The second channel attention module includes an average pooling layer, a maximum pooling layer and a fully connected layer. The channel attention operation is implemented in the frequency domain, where the fully connected layer is implemented through a multi-layer perceptron MLP layer.

[0087] In some embodiments, as Figure 7 As shown, the input image and the prior information are input into the pre-fusion module of each Transformer layer in the Transformer module, and the output of the preliminary fusion feature further includes:

[0088] 511. Input the prior information into a second channel attention module and output a channel attention feature; the second channel attention module includes a fourth average pooling layer and a fourth maximum pooling layer; the channel attention feature includes the fused average pooling feature and the maximum pooling feature;

[0089] 512. Input the channel attention feature into a fourth multi-layer perceptron and multiply it by the prior information to output the prior channel feature;

[0090] 513. Input the fitted noise image into a convolutional layer and output a fitted noise feature;

[0091] 514. Concatenate the priori channel features and the fitted noise features to obtain preliminary fusion features.

[0092] This embodiment assigns different attention weights to different frequency components through the second channel attention module to achieve the purpose of capturing key frequency information and avoid the problem of spectral distortion.

[0093] 106. Input the fitted noise image into a diffusion model, and output a reconstructed difference image between the real image and the multispectral image;

[0094] Corresponding to step 104, in some embodiments, during the inverse generation process, this embodiment removes a certain amount of Gaussian noise from the fitted noise image at each time step t. Through this process, starting from standard Gaussian noise, the noise is gradually removed through the learned inverse diffusion process to generate a reconstructed difference image with the same distribution as the original difference image. The reconstructed difference image is the difference image after the Gaussian noise is removed.

[0095] 107. Add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0096] In this embodiment, the reconstructed difference image obtained in step 106 is superimposed on the multispectral image acquired in step 101 to obtain a high-resolution multispectral image.

[0097] 108. Calculating a loss value of a remote sensing image pan-sharpening model based on a difference image between the high-resolution multispectral image and the real image;

[0098] See also Figure 8 ,like Figure 8 As shown, the remote sensing image full-color sharpening model includes a feature extraction module, a feature fusion module, a diffusion model and a Transformer module, wherein the diffusion model is represented by superimposing Gaussian noise, which is not shown; on the one hand, the full-color image PAN and the upsampled multispectral image LMS↑4 are input into the feature extraction module, and after passing through the feature fusion module, they are input into the Transformer model together with the time information T as prior information; on the other hand, the difference image between the real image GT and the upsampled multispectral image LMS↑4 is added with noise and input into the Transformer model together; the Transformer model generates a reconstructed difference image between the real image GT and the upsampled multispectral image LMS↑4; and then the reconstructed difference image is superimposed with the upsampled multispectral image LMS↑4 to obtain the final high-resolution multispectral image. Generate high-resolution multispectral images Corresponding to the real input image.

[0099] 109. Update the model parameters of the remote sensing image pan-sharpening model through gradient according to the loss value.

[0100] In one possible implementation, the method for obtaining the pan-sharpening model loss of the remote sensing image based on the difference image between the high-resolution multispectral image and the real image can specifically be to perform cross-entropy loss calculation on the difference image between the high-resolution multispectral image and the real image to obtain the model loss.

[0101] It should be understood that the above-mentioned process of obtaining the model loss is only illustrative. If there are other intermediate features in the model training process (such as intermediate coding features of one or more coding layers in the encoding process or features in the decoding process), the model loss can also be obtained based on the intermediate features and the above-mentioned features, or based on the intermediate features and the above-mentioned coding features, labels and image processing results. The model loss is obtained, which will not be described one by one in this embodiment.

[0102] When adjusting the model parameters of the remote sensing image full-color sharpening model based on the model loss, the purpose is to minimize the model loss so that the remote sensing image full-color sharpening model gradually converges. When the number of times the remote sensing image full-color sharpening model parameters are adjusted reaches a preset number, for example, 1000 times, 10000 times, etc., the remote sensing image full-color sharpening model can be considered to have converged. It can also be when the model loss gradually tends to a fixed value, such as zero, or is less than a preset value, such as 0.05, 0.01, etc., the model can be considered to have converged, and the trained remote sensing image full-color sharpening model can be obtained.

[0103] See also Figure 9 , the embodiment of the present application also proposes a remote sensing image full color sharpening method, the method comprising:

[0104] 111. Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0105] 112. Input the target panchromatic image, the target multispectral image, and Gaussian noise into a trained remote sensing image pan-sharpening model, and output a reconstructed difference image of the target remote sensing image;

[0106] 113. Add the reconstructed difference image to the target multispectral image to obtain a high-resolution target multispectral image.

[0107] By adopting the above-mentioned remote sensing image full-color sharpening method of the present application, since the model used in the remote sensing image full-color sharpening process is a model trained using the aforementioned remote sensing image full-color sharpening model training method, therefore, when the above-mentioned remote sensing image full-color sharpening model is used during the sharpening process, the accuracy and precision of the extracted features can be improved. Therefore, when image sharpening is performed based on the extracted features, the generation accuracy of high-resolution multispectral images can be effectively improved.

[0108] See also Figure 10 , an embodiment of the present application provides a remote sensing image pan-sharpening model training system:

[0109] An acquisition module 301 is configured to acquire multiple sets of training remote sensing images, each set of training remote sensing images including a panchromatic image, a multispectral image, and a real image;

[0110] A feature extraction module 302 is configured to input the panchromatic image and the multispectral image into a feature extraction module and output panchromatic features and spectral features;

[0111] A feature fusion module 303 is configured to input the panchromatic feature and the spectral feature into a feature fusion module and output a panchromatic-spectral fusion feature;

[0112] a diffusion model processing module 304 for inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model and outputting a diffusion noise image; and for inputting the fitted noise image into the diffusion model and outputting a reconstructed difference image between the real image and the multispectral image;

[0113] Transformer processing module 305, used for processing the diffusion-noised image and the panchromatic-spectral fusion feature to generate a priori-noised image;

[0114] An image processing module 306 is configured to add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image;

[0115] The model training module 307 is used to calculate the loss value of the remote sensing image full-color sharpening model based on the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image full-color sharpening model based on the loss value; the remote sensing image full-color sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

[0116] See also Figure 11 , an embodiment of the present application provides a remote sensing image pan-sharpening system:

[0117] An input module 401 is configured to acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image;

[0118] A first processing module 402 is configured to input the target panchromatic image and the target multispectral image into a trained remote sensing image pan-sharpening model, and output a reconstructed difference image of the target remote sensing image;

[0119] A second processing module 403 is configured to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image;

[0120] The output module 404 is configured to display the high-resolution target multispectral image.

[0121] The input module 401 can be a remote sensing satellite, an unmanned aerial vehicle (UAV), or other high-altitude platform, as well as a multispectral camera or remote sensing equipment, and can acquire high-resolution panchromatic images of the target and low-resolution multispectral images of the target. The output module 404 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode), and can display high-resolution multispectral images of the target.

[0122] Each module in the above-mentioned remote sensing image full-color sharpening model training system and remote sensing image full-color sharpening system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the device embodiment in this application corresponds to the above-mentioned method embodiment. The specific principles in the system embodiment can be found in the contents of the above-mentioned method embodiment, which will not be repeated here.

[0123] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image pan-sharpening model training method, characterized in that: The method comprises: Acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image; Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features; Inputting the panchromatic image features and the spectral image features into a feature fusion module, and outputting panchromatic-spectral fusion features; Inputting a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model, and outputting a diffusion noise image; Input the diffuse noise image and the panchromatic-spectral fusion feature into the Transformer module, and output a fitted noisy image with panchromatic-spectral information; Inputting the fitted noisy image into a diffusion model, and outputting a reconstructed difference image between the real image and the multispectral image; Adding the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image; Calculating the loss value of a remote sensing image pan-sharpening model based on a difference image between the high-resolution multispectral image and the real image; the remote sensing image pan-sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module; According to the loss value, the model parameters of the remote sensing image pan-sharpening model are updated through gradients.

2. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: After obtaining the multiple sets of training remote sensing images, the method further includes performing data preprocessing on the multiple sets of training remote sensing images; the data preprocessing includes calculating the to-be-processed training remote sensing images corresponding to the corresponding training remote sensing images based on the mean and standard deviation of each training remote sensing image; For each to-be-processed training remote sensing image in the plurality of sets of training remote sensing images, data enhancement is performed on each to-be-processed training remote sensing image; the data enhancement includes random flipping, random blurring, and random color adjustment.

3. The remote sensing image pan-sharpening model training method according to claim 1 or 2, characterized in that: After obtaining multiple sets of training remote sensing images, the method further includes upsampling the multispectral images of each set of training remote sensing images.

4. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features includes: In the first stage, the panchromatic image and the multispectral image are input into a first feature extraction unit with a spatial attention module, and a first-stage panchromatic image feature map and a first-stage multispectral image feature map are output; the first feature extraction unit includes a first average pooling layer, a second average pooling layer, a first maximum pooling layer, a second maximum pooling layer, a first multilayer perceptron, and a first spatial attention module; In the second stage, the panchromatic image feature map and the multispectral image feature map of the first stage are input into a second feature extraction unit with a channel attention module, and the second stage panchromatic image feature map and the second stage multispectral image feature map are output; the second feature extraction unit includes a third average pooling layer, a third maximum pooling layer, a second multilayer perceptron and the first channel attention module; The second-stage panchromatic image feature map is used to indicate panchromatic image features, and the second-stage multispectral image feature map is used to indicate spectral image features.

5. The remote sensing image pan-sharpening model training method according to claim 4, characterized in that: Inputting the panchromatic image and the multispectral image into a feature extraction module, and outputting panchromatic image features and spectral image features specifically includes: Inputting the panchromatic image into a first average pooling layer and a first maximum pooling layer, respectively, inputting the multispectral image into a second average pooling layer and a second maximum pooling layer, respectively, and outputting the maximum features and average features of the panchromatic image and the multispectral image; After interacting with the maximum features and average features of the panchromatic image and the multispectral image through a first spatial attention module, the maximum features and the average features are input into a first multi-layer perceptron, and the spatial feature weights of the panchromatic image and the multispectral image are output; Multiplying the spatial feature weight of the panchromatic image by the panchromatic image and performing residual connection to obtain a first-stage panchromatic image feature map; Multiplying the spatial feature weights of the multispectral image by the multispectral image and performing residual connection to obtain a first-stage multispectral image feature map; After the first-stage panchromatic image feature map and the first-stage multispectral image feature map interact with each other through the first channel attention module, they are input into the third maximum pooling layer and the third average pooling layer respectively, and the maximum feature and average feature of the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map are output; Inputting the maximum feature and the average feature after the interaction between the first-stage panchromatic image feature map and the first-stage multispectral image feature map into a second multi-layer perceptron, and outputting the channel feature weights of the first-stage panchromatic image feature map and the first-stage multispectral image feature map; Multiplying the first-stage full-color image feature map by the channel feature weight of the first-stage full-color image feature map and performing residual connection to obtain the second-stage full-color image feature map; The first-stage multispectral image is multiplied by the channel feature weight of the first-stage multispectral image and then a residual connection is performed to obtain a second-stage multispectral image feature map.

6. The remote sensing image pan-sharpening model training method according to claim 1, characterized in that: Inputting the diffuse noise image and the panchromatic-spectral fusion feature into a Transformer module and outputting a priori noisy image includes: Using the panchromatic-spectral fusion feature as prior information; using the diffuse noise image as an input image; Inputting the input image and the prior information into a pre-fusion module of each Transformer layer in the Transformer module, and outputting preliminary fusion features; Extracting the key vector K and value vector V of the prior information, and extracting the query vector Q of the preliminary fusion feature; Performing Fourier transform on the query vector Q, key vector K, and value vector V, multiplying the query vector Q by the key vector K in the frequency domain through a cross-attention mechanism, performing a softmax operation, and then multiplying the query vector Q by the value vector V and performing an inverse Fourier transform to obtain a hierarchical fusion feature of the diffuse noise image and the panchromatic-spectral fusion feature; The hierarchical fusion features are input into the third multi-layer perceptron layer of the corresponding level, and the output result of the third multi-layer perceptron of the corresponding level is used as the input data of the next Transformer layer in the Transformer module until the third multi-layer perceptron of the last level outputs the prior noisy image.

7. The remote sensing image pan-sharpening model training method according to claim 6, characterized in that: The input image and the prior information are input into the pre-fusion module of each Transformer layer in the Transformer module, and the output preliminary fusion features include: Inputting the prior information into a second channel attention module and outputting a channel attention feature; the second channel attention module includes a fourth average pooling layer and a fourth maximum pooling layer; the channel attention feature includes the fused average pooling feature and the maximum pooling feature; Inputting the channel attention feature into a fourth multi-layer perceptron and multiplying it by the prior information, outputting the prior channel feature; Inputting the fitted noise image into a convolutional layer and outputting fitted noise features; The prior channel features are concatenated with the fitted noise features to obtain preliminary fusion features.

8. A remote sensing image pan-sharpening method, characterized in that: The method comprises: Acquire a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image; Inputting the target panchromatic image, the target multispectral image, and Gaussian noise into a pan-sharpening model, wherein the pan-sharpening model is trained by the remote sensing image pan-sharpening model training method according to any one of claims 1 to 7, and outputting a reconstructed difference image of the target remote sensing image; The reconstructed difference image is added to the target multispectral image to obtain a high-resolution target multispectral image.

9. A remote sensing image pan-sharpening model training system, characterized in that: The method is based on a remote sensing image pan-sharpening model training method according to any one of claims 1 to 7, wherein the training system comprises: An acquisition module is used to acquire multiple sets of training remote sensing images, each set of training remote sensing images includes a panchromatic image, a multispectral image, and a real image; A feature extraction module, configured to input the panchromatic image and the multispectral image into the feature extraction module and output panchromatic features and spectral features; A feature fusion module, configured to input the panchromatic feature and the spectral feature into a feature fusion module and output a panchromatic-spectral fusion feature; a diffusion model processing module, configured to input a difference image between the real image and the multispectral image and Gaussian noise into a diffusion model and output a diffusion noise image; and to input the fitted noisy image into the diffusion model and output a reconstructed difference image between the real image and the multispectral image; A Transformer processing module, configured to process the diffuse noise image and the panchromatic-spectral fusion feature to generate a priori noisy image; An image processing module, configured to add the reconstructed difference image to the multispectral image to obtain a high-resolution multispectral image; The model training module is used to calculate the loss value of the remote sensing image full-color sharpening model based on the difference image between the high-resolution multispectral image and the real image; and to update the model parameters of the remote sensing image full-color sharpening model based on the loss value; the remote sensing image full-color sharpening model includes a feature extraction module, a feature fusion module, a diffusion model, and a Transformer module.

10. A remote sensing image full color sharpening system, characterized in that: The sharpening system adopts the remote sensing image pan-sharpening model trained by the training system as claimed in claim 9, and includes: An input module is used to obtain a target remote sensing image and Gaussian noise, wherein the target remote sensing image includes a high-resolution target panchromatic image and a low-resolution target multispectral image; A first processing module is configured to input the target panchromatic image and the target multispectral image into a trained remote sensing image pan-sharpening model, and output a reconstructed difference image of the target remote sensing image; a second processing module, configured to add the reconstructed difference image to the target multispectral image to output a high-resolution target multispectral image; The output module is used to display the high-resolution target multispectral image.

Images