Panchromatic sharpening method based on progressive expansion frame

Abstract

The invention discloses a panchromatic sharpening method based on a progressive expansion frame. The panchromatic sharpening method comprises the following steps: acquiring a to-be-optimized image; the to-be-optimized image is input to a variation optimization model, a clear multispectral image is obtained, the variation optimization model is obtained through training of a training set, the training set comprises the multispectral image and a panchromatic image, the variation optimization model is used for mutually modulating the multispectral image and the corresponding panchromatic image under different scales, and the multispectral image and the panchromatic image are obtained. And supplementing detail information of the multispectral image and retaining spectral features to obtain the clear multispectral image. According to the method, the quality of the fused image is effectively improved on the simulated data set, and the method has better spectral information retention capability and space detail supplementation capability.

Description

Technical Field

[0001] The present invention belongs to the technical field of panchromatic sharpening, and particularly relates to a panchromatic sharpening method based on a progressive unfolding framework. Background Technique

[0002] With the progress of remote sensing technology, multi-spectral images have been widely used in tasks such as land cover classification, target recognition, and target detection. However, due to inherent physical hardware limitations, high-resolution multi-spectral images cannot be directly obtained. To balance spectral and spatial features, satellites usually equipped with two different types of sensors to capture low-resolution multi-spectral images and high-resolution single-band panchromatic images of the same scene respectively. The panchromatic sharpening task emerged as the times require, aiming to fuse low-resolution multi-spectral images and high-resolution panchromatic images and generate high-resolution multi-spectral images with rich spatial details and accurate spectral information. Such high-quality multi-spectral images can significantly improve the accuracy of downstream tasks.

[0003] Currently, the methods for solving the panchromatic sharpening task are mainly divided into four categories, including three traditional methods and one deep learning method, which are specifically as follows:

[0004] The first category is the component replacement method, whose main idea is to map the upsampled low-resolution multi-spectral image to the transform domain, replace the intensity component with the panchromatic image, and then use the inverse transform to obtain the high-resolution multi-spectral image. This method effectively restores spatial information, but is prone to serious spectral distortion.

[0005] The second category is the multi-resolution analysis method, whose main idea is to use multi-scale decomposition methods or spatial filtering methods to obtain spatial details, and then inject the spatial details into the upsampled low-resolution multi-spectral image to obtain the target image. This method is easy to retain color information, but some spatial details will be lost due to repeated transformations.

[0006] The third category is the variational optimization method, whose main idea is to regard the panchromatic sharpening task as an ill-posed problem, use the degradation model and prior constraints to construct an energy equation, and iteratively optimize the quality of the target image. This method is conducive to finding the global optimality, but its prior is usually set artificially and it is difficult to adapt to various complex scenarios.

[0007] The fourth category is the deep learning method, whose main idea is to use network frameworks such as convolutional neural networks, Transformers, and generative adversarial networks to adaptively extract the features of the source images and generate high-quality target images. This method has strong non-linear fitting ability and fast inference ability, but it lacks interpretability and highly depends on the quantity and quality of the training data.

[0008] Although the models in the field of panchromatic sharpening have made significant progress, the existing technologies still have three main drawbacks.

[0009] First, the traditional methods have varying degrees of spectral feature distortion and loss of spatial details. Although deep learning methods can better balance spectral information and spatial information, they have the characteristics of a black box and are highly dependent on the quality and quantity of training data. Once the distribution of the test images does not conform to the training data, the model will experience serious performance degradation.

[0010] Second, both traditional methods and deep learning methods usually adopt a preprocessing step of directly upsampling the low-resolution multispectral image by four times to match the spatial resolution of the panchromatic image, and then perform subsequent fusion. However, this operation is too simple and ignores the serious feature loss problem during the direct upsampling process of the low-resolution multispectral image, thereby affecting subsequent operations.

[0011] Finally, when using the panchromatic image to supplement the details of the multispectral image, existing deep learning models usually adopt fixed convolutional kernels and global attention mechanisms. Fixed convolutional kernels will cause the model to lack flexibility in processing different spatial details, may not be able to adapt to local transformations, and it is difficult to accurately capture the detail information in different regions of the image. Although the global attention mechanism can improve the utilization of context information and enhance global relationships, its computational complexity is relatively high, and it may cause irrelevant or redundant regions to be overly concerned. It is difficult to capture the features and details of small regions only by using global attention. Therefore, there is still room for development in the existing technology for adaptive local fusion.

[0012] To solve the above problems, the present invention proposes a panchromatic sharpening method based on a progressive unfolding framework, which decomposes the panchromatic sharpening task into a three-stage progressive multispectral image restoration task guided by the panchromatic image. Summary of the Invention

[0013] To solve the above technical problems, the present invention proposes a panchromatic sharpening method based on a progressive unfolding framework, which can solve the problems of spectral feature distortion and insufficient spatial details of the fused image and improve the quality of the fused image.

[0014] The present invention provides a panchromatic sharpening method based on a progressive unfolding framework, including:

[0015] Obtain the image to be optimized;

[0016] Input the image to be optimized into a variational optimization model to obtain a clear multispectral image, where the variational optimization model is obtained by training with a training set, the training set includes multispectral images and panchromatic images, and the variational optimization model is used to mutually modulate the multispectral images at different scales with the corresponding panchromatic images, supplement the detail information of the multispectral images and retain spectral features to obtain the clear multispectral image.

[0017] Optionally, obtaining the training set includes:

[0018] Obtaining an original multispectral image and a panchromatic image;

[0019] Performing downsampling operations on the original multispectral image and the panchromatic image to obtain processed images;

[0020] Segmenting the processed images to obtain the training set.

[0021] Optionally, the variational optimization model includes: a number of prior learning modules and a number of image update modules;

[0022] The prior learning module is used to restrict the restoration direction of the multispectral image, utilize the panchromatic image to guide the restoration of the high-frequency details and enhancement of the low-frequency structure of the multispectral image, and denoise the enhanced multispectral image;

[0023] The image update module is used to update the multispectral image.

[0024] Optionally, before obtaining the clear multispectral image, it further includes:

[0025] Obtaining a panchromatic image, and performing spectral super-resolution on the panchromatic image by using convolution and activation functions to obtain a first feature map;

[0026] Performing wavelet sampling on the first feature map to obtain a second feature map and a first high-frequency image;

[0027] Performing wavelet sampling on the second feature map to obtain a third feature map and a second high-frequency image.

[0028] Optionally, obtaining the clear multispectral image includes:

[0029] Obtaining a first blurred multispectral image;

[0030] Performing a first-scale transformation on the first blurred multispectral image in combination with the third feature map to obtain a first clear multispectral image;

[0031] Performing high-frequency modulation and inverse wavelet transform on the first clear multispectral image in combination with the second high-frequency image to obtain a second blurred multispectral image;

[0032] Performing a second-scale transformation on the second blurred multispectral image in combination with the second feature map to obtain a second clear multispectral image;

[0033] Performing high-frequency modulation and inverse wavelet transform on the second clear multispectral image in combination with the first high-frequency image to obtain a third blurred multispectral image;

[0034] Perform a third-scale transformation on the third blurred multi-spectral image in combination with the first feature map to obtain the clear multi-spectral image.

[0035] Optionally, performing a scale transformation on a blurred multi-spectral image in combination with a feature map includes:

[0036] Input the blurred multi-spectral image into the prior learning module and the image update module in sequence for multiple iterations. The number of iterations in each stage is set to 3 to obtain the clear multi-spectral image.

[0037] Optionally, the method for inputting the blurred multi-spectral image into the prior learning module and the image update module in sequence to obtain the clear multi-spectral image is:

[0038] ;

[0039] ;

[0040] where is the th iteration, is the scale transformation process ( = 1, 2, 3), is an auxiliary variable to assist in updating the multi-spectral image, is the proximal operation of the prior . The whole process is implicitly implemented by the prior learning module here, and are step sizes, is the clear multi-spectral image, is the blur kernel, is the transpose of the blur kernel, is the blurred multi-spectral image, is the scale parameter.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] 1. The present invention cleverly combines the advantages of variational optimization traditional methods and deep learning methods, not only enhancing the interpretability and transparency of the network, but also improving the fusion performance of the algorithm. The invention designs a progressive fusion strategy based on wavelet transform to gradually increase the details of the multi-spectral image at three spatial resolutions, alleviating the information loss problem caused by directly upsampling the low-resolution multi-spectral image. The invention proposes a prior learning module with multi-scale local cross-attention as the core, which effectively supplements the detail information of the multi-spectral image with the panchromatic image and retains the spectral characteristics.

[0043] 2. The present invention effectively improves the quality of the fused image on the simulated data set, and has better spectral information retention ability and spatial detail supplementation ability. At the same time, the present invention also has sufficient generalization ability in the actual application of real images, can adapt to complex real scenes, and can well balance spectral information and spatial information in different scenes such as vegetation, roads, oceans, and buildings. Brief Description of the Drawings

[0044] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0045] Figure 1 is a flowchart of the pan-sharpening method based on the progressive unfolding framework according to the embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the network framework according to the embodiment of the present invention;

[0047] Figure 3 is a schematic diagram of the experimental results according to the embodiment of the present invention. Detailed Embodiments

[0048] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the drawings and combine the embodiments to detail this application.

[0049] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0050] This embodiment proposes a pan-sharpening method based on the progressive unfolding framework, as Figure 1 shown, which specifically includes the following steps:

[0051] Obtain the image to be optimized;

[0052] Input the image to be optimized into the variational optimization model to obtain a clear multi-spectral image. Among them, the variational optimization model obtains the optimal parameters through the training set, and the training set includes multi-spectral images and panchromatic images. The variational optimization model is used to mutually modulate the multi-spectral images at different scales with the corresponding panchromatic images, supplement the detail information of the multi-spectral images and retain the spectral features to obtain clear multi-spectral images.

[0053] Furthermore, obtaining the training set includes:

[0054] Obtain the original multi-spectral image and panchromatic image;

[0055] Perform downsampling operations on the original multispectral image and the panchromatic image to obtain the processed images;

[0056] Segment the processed images to obtain the training set.

[0057] Specifically, in this embodiment, the WorldView-3 dataset is used. The original multispectral image and the panchromatic image are downsampled by a factor of four to obtain simulated data. The original multispectral image is used as the reference image for calculating the loss. Then, the low-resolution multispectral image is randomly cut into images of size 16×16, and the corresponding panchromatic image and reference image are cut into images of size 64×64 to obtain the training data.

[0058] After obtaining the processed training dataset, it is input into the network in batches for training and parameter update. The loss function of the network uses the mean absolute error (MAE), which can reduce the sensitivity to noise. The formula is as follows:

[0059] (1);

[0060] Where, is the mean absolute error loss function, is the clear multispectral image finally output at the third scale, is the corresponding reference image.

[0061] When the loss no longer decreases, save the converged network parameters, perform fusion tests on unknown data, and conduct reasonable evaluations through quantitative metrics and qualitative vision.

[0062] Furthermore, the variational optimization model includes: a number of prior learning modules and a number of image update modules;

[0063] The prior learning module is used to restrict the restoration direction of the multispectral image, use the panchromatic image to guide the restoration of the high-frequency details and enhance the low-frequency structure of the multispectral image, and denoise the enhanced multispectral image;

[0064] The image update module is used to update the multispectral image.

[0065] Specifically, the network framework constructed in this embodiment is based on the iterative solution process of the variational optimization model. Other model-driven deep learning methods assume that the low-resolution multispectral image is obtained by blurring and downsampling the high-resolution multispectral image. Different from the existing assumptions, in this embodiment, in order to reduce the information loss caused by upsampling the low-resolution multispectral image, the fusion task at a single spatial resolution is decomposed into image restoration tasks at three spatial resolutions, assuming that the blurred multispectral image at each scale is restored to a clear multispectral image under the guidance of the panchromatic image. Assume that the different scales are respectively represented as , at the scale, the panchromatic image is represented as , the blurred multispectral image is represented as , the clear multispectral image is represented as , the Gaussian blur kernel is represented as , then the following variational optimization energy equation can be obtained:

[0066] (2);

[0067] Among them, represents the denoising prior of the multispectral image guided by the panchromatic image, represents the proportionality parameter. In the traditional variational optimization method, this prior is artificially designed and relies on limited domain prior knowledge, making it difficult to adapt to complex real-world scenarios. Therefore, in this embodiment, in order to improve the prior representation ability, a deep learning method is used to adaptively learn complex prior knowledge from the training data. This energy equation can be solved by the semi-quadratic splitting method. By introducing an auxiliary variable , the equation is decomposed into an auxiliary variable estimation process and a multispectral image update process. Among them, the auxiliary variable estimation process is also called the prior learning process, and the obtained equation is as follows:

[0068] ;

[0069] Among them, represents the proportionality parameter, represents the th iteration. Assuming and represent the step size, the specific iteration formula is as follows:

[0070] ;

[0071] ;

[0072] Furthermore, before obtaining the clear multispectral image, it also includes:

[0073] Obtain the panchromatic image, perform spectral super-resolution on the panchromatic image using convolution and activation functions, and obtain the first feature map;

[0074] Perform wavelet sampling on the first feature map to obtain the second feature map and the first high-frequency image;

[0075] Perform wavelet sampling on the second feature map to obtain the third feature map and the second high-frequency image.

[0076] Furthermore, obtaining the clear multispectral image includes:

[0077] Obtain the first blurred multispectral image;

[0078] Perform a first-scale transformation on the first blurred multispectral image combined with the third feature map to obtain a first clear multispectral image;

[0079] Perform high-frequency modulation and inverse wavelet transform on the first clear multispectral image combined with the second high-frequency image to obtain a second blurred multispectral image;

[0080] Perform a second-scale transformation on the second blurred multispectral image combined with the second feature map to obtain a second clear multispectral image;

[0081] Perform high-frequency modulation and inverse wavelet transform on the second clear multispectral image combined with the first high-frequency image to obtain a third blurred multispectral image;

[0082] Perform a third-scale transformation on the third blurred multispectral image combined with the first feature map to obtain a clear multispectral image.

[0083] Furthermore, performing a scale transformation on a blurred multispectral image combined with a feature map includes:

[0084] Input the blurred multispectral image into a prior learning module and an image update module in sequence for multiple iterations to obtain a clear multispectral image.

[0085] Specifically, the network invented in this embodiment is divided into three major stages according to three scales , and each major stage contains small stages to iteratively recover the multispectral image. Each small stage contains a prior learning module and a target image update module, corresponding to formulas (3) and (4) respectively. This network is a bidirectional network. The high-resolution panchromatic image first performs spectral super-resolution using convolution and activation functions, and the obtained feature map is used to guide image restoration. This feature map is then downsampled twice through wavelet transform, retaining the high-frequency feature maps at the middle two scales, while the low-frequency feature maps at the two scales are respectively used for and prior guidance learning. At the end of each major stage, after obtaining the multispectral image with enhanced details at this scale, use the details of this image to perform interactive modulation on the high frequencies of the three panchromatic images at this scale, so that the four feature maps are flexibly aligned and then perform inverse wavelet transform upsampling to improve the ability to retain spectra and reduce detail redundancy. To better capture local details, this embodiment proposes a prior module centered on multi-scale local cross-attention, which generates a local adaptive kernel from the fused features of the multispectral image and the panchromatic image, used to supplement the detail information of the multispectral image and retain spectral features, and realizes multi-scale attention enhancement by combining adaptive kernels of different sizes. The specific network framework is as Figure 2 shown.

[0086] The following combines the attachedFigure 3 The detailed description of this embodiment is as follows:

[0087] This embodiment is trained and tested on the WorldView-3 dataset. The training batch size is set to 8, the learning rate is set to 0.0005, and the number of training image pairs is 9714. The experimental environment is Windows 11, 3.4 GHZ AMD Ryzen 5950X, NVIDIA GeForce RTX 3090 GPU, and Python 3.9. It is compared with 9 other pan-sharpening algorithms, and the experimental results are as Figure 3 shown. It can be seen that only this embodiment successfully retains the red color and clear edge details in the reference image, indicating the effectiveness of this embodiment in improving the quality of the fused image.

[0088] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the technical field of the present application within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A panchromatic sharpening method based on a progressive unfolding framework, characterized in that, Including: Obtain the image to be optimized; Input the image to be optimized into the variational optimization model to obtain a clear multispectral image, where the variational optimization model is obtained by training with a training set, the training set includes multispectral images and panchromatic images, and the variational optimization model is used to perform mutual modulation on the multispectral images at different scales with the corresponding panchromatic images, supplement the detail information of the multispectral images and retain the spectral characteristics to obtain the clear multispectral image.

2. The panchromatic sharpening method based on the progressive unfolding framework according to claim 1, wherein Obtaining the training set includes: Obtain the original multispectral image and panchromatic image; Perform downsampling operations on the original multispectral image and panchromatic image to obtain the processed images; Segment the processed images to obtain the training set.

3. The panchromatic sharpening method based on a progressive unfolding framework according to claim 1, wherein, The variational optimization model includes: a number of prior learning modules and a number of image update modules; The prior learning module is used to restrict the restoration direction of the multispectral image, use the panchromatic image to guide the restoration of the high-frequency details and enhancement of the low-frequency structure of the multispectral image, and denoise the enhanced multispectral image; The image update module is used to perform one iteration update on the multispectral image.

4. The panchromatic sharpening method based on the progressive unfolding framework according to claim 3, characterized in that Before obtaining the clear multispectral image, it also includes: Obtain the panchromatic image, perform spectral super-resolution on the panchromatic image using convolution and activation functions to obtain the first feature map; Perform wavelet sampling on the first feature map to obtain the second feature map and the first high-frequency image; Perform wavelet sampling on the second feature map to obtain the third feature map and the second high-frequency image.

5. The panchromatic sharpening method based on the progressive unfolding framework according to claim 4, wherein Obtaining the clear multispectral image includes: Obtain the first blurred multispectral image; Perform a first-scale transformation on the first blurred multispectral image combined with the third feature map to obtain the first clear multispectral image; Perform high-frequency modulation and inverse wavelet transform on the first clear multispectral image combined with the second high-frequency image to obtain the second blurred multispectral image; Perform a second-scale transformation on the second blurred multispectral image combined with the second feature map to obtain the second clear multispectral image; Perform high-frequency modulation and inverse wavelet transform on the second clear multispectral image combined with the first high-frequency image to obtain the third blurred multispectral image; Perform a third-scale transformation on the third blurred multispectral image combined with the first feature map to obtain the clear multispectral image.

6. The panchromatic sharpening method based on the progressive unfolding framework according to claim 5, characterized in that, Performing a scale transformation on the blurred multispectral image combined with the feature map includes: Input the blurred multispectral image into the prior learning module and the image update module in sequence for multiple iterations, and set the number of iterations in each stage to 3 to obtain the clear multispectral image.

7. The panchromatic sharpening method based on the progressive unfolding framework according to claim 6, wherein The method of inputting the blurred multispectral image into the prior learning module and the image update module in sequence to obtain the clear multispectral image is: ； ； wherein, is the th iteration, is the scale transformation process ( = 1, 2, 3), is an auxiliary variable to assist in updating the multispectral image, is the proximal operation of the prior , and the whole process is implicitly implemented by the prior learning module here, and are step sizes, is the clear multispectral image, is the blur kernel, is the transpose of the blur kernel, is the blurred multispectral image, is the scale parameter.

Images