Panchromatic Sharpening Method, Device, Electronic Device and Medium for Satellite Remote Sensing Images

Abstract

An embodiment of the present invention relates to the field of remote sensing image processing technology, and provides a panchromatic sharpening method, device, electronic device and medium for satellite remote sensing images. The method includes: obtaining satellite remote sensing images of a panchromatic modality and a multispectral modality to be processed; performing upsampling processing on the multispectral modality image until the target output size of the target panchromatic sharpened image is reached, to obtain a first satellite remote sensing image of the multispectral modality; performing multi-scale feature extraction on the panchromatic modality image to obtain different scale features of the satellite remote sensing image of the panchromatic modality; performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic modality image with the same scale as the first satellite remote sensing image to obtain high-frequency features and low-frequency features of different scales; performing feature fusion on the high-frequency and low-frequency features with the same scale, and performing image reconstruction to obtain the target panchromatic sharpened image. Thereby, the detail richness and visual coordination of the reconstructed image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method, device, electronic equipment and medium for panchromatic sharpening of satellite remote sensing images. Background Art

[0002] Panchromatic sharpening is one of the key processing links of satellite images, which has great application value and broad prospects. Panchromatic sharpening technology complements and fuses panchromatic and multispectral remote sensing images, overcomes the shortcomings of insufficient information from single-modal remote sensing images, and maximizes the advantages of each modality, which plays an important role in downstream remote sensing tasks such as change detection, recognition, and detection.

[0003] Under constraints, full-color images have high spatial resolution, but can only provide single-channel band information. Multispectral images have rich spectral information, but their spatial resolution is poor. In the fusion process, it is necessary to combine the advantages of different modalities, fully extract the features of each modality, fuse complementary features, and remove redundant features. Existing research is based on deep learning to model the mapping relationship between the two modalities. Modal features are proposed through convolutional neural networks, and Transformer is used for long-distance feature interaction to model the correlation of cross-modal features. Some researchers also convert the image to the frequency domain first, and then model the spectral relationship through convolutional neural networks. In the reconstruction process, the loss function of these methods uses L1 loss or L2 loss.

[0004] A common approach is to use the global modeling capabilities of the Transformer mechanism to interact with cross-modal features or use transformation tools to convert images to the frequency domain, and combine the frequency domain and spatial domain to learn spectral distribution features and spatial distribution features. However, in the process of neural network learning, the network easily learns low-frequency features and tends to ignore high-frequency features. Some high-frequency information will be smoothed by the network or lost as an outlier. The above methods do not selectively enhance the high-frequency and low-frequency parts of image features differently, which will result in the lack of high-frequency texture in the reconstructed image.

[0005] When using L2 loss, the network tends to smooth out points with larger pixel values, which are often high-frequency details in remote sensing images. When only L1 loss and L2 loss are used as reconstruction losses, the reconstructed image is clear but does not conform to human eye perception and the overall look and feel is not harmonious. Summary of the invention

[0006] The present invention provides a method, device, electronic device and medium for panchromatic sharpening of satellite remote sensing images, which are used to solve the defects of panchromatic sharpening and reconstructing remote sensing images in the prior art, such as lack of high-frequency texture, incompatibility with human eye perception and overall inharmonious appearance, so as to improve the detail richness and visual coordination of fused images.

[0007] The present invention provides a panchromatic sharpening method for satellite remote sensing images, including:

[0008] Obtaining a satellite remote sensing image in panchromatic mode and a satellite remote sensing image in multispectral mode to be processed;

[0009] Performing upsampling processing on the satellite remote sensing image in multispectral mode until reaching the target output size of the target panchromatic sharpened image, to obtain a first satellite remote sensing image in multispectral mode with the target output size;

[0010] Performing multi-scale feature extraction on the satellite remote sensing image in panchromatic mode to obtain different scale features of the satellite remote sensing image in panchromatic mode;

[0011] Performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image in panchromatic mode with the same scale as the first satellite remote sensing image, to obtain high-frequency features of different scales and low-frequency features of different scales;

[0012] Fusing the high-frequency features and low-frequency features with the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0013] In a possible implementation manner, the method further includes:

[0014] Performing convolution operation on the satellite remote sensing image in panchromatic mode to extract local features of the satellite remote sensing image in panchromatic mode;

[0015] Performing multiple pooling operations on the satellite remote sensing image in panchromatic mode to extract multi-scale features of the satellite remote sensing image in panchromatic mode;

[0016] Based on the local features and the multi-scale features, obtaining different scale features of the satellite remote sensing image in panchromatic mode.

[0017] In a possible implementation manner, the method further includes:

[0018] Obtaining the upsampled image features of the first satellite remote sensing image;

[0019] Performing dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the satellite remote sensing image in panchromatic mode with the same scale as the first satellite remote sensing image, to obtain high-frequency information of different scales and low-frequency information of different scales;

[0020] Performing frequency-selective enhancement processing on the high-frequency information of different scales and the low-frequency information of different scales respectively, to obtain high-frequency features of different scales and low-frequency features of different scales.

[0021] In a possible implementation, the method further includes:

[0022] Performing frequency fusion on the high-frequency information and low-frequency information of the same scale based on a preset frequency fusion strategy to obtain frequency information features of different scales after fusion.

[0023] In a possible implementation, the method further includes:

[0024] Performing frequency-selective enhancement processing on the frequency information features of different scales to obtain high-frequency features of different scales and low-frequency features of different scales.

[0025] In a possible implementation, the method further includes:

[0026] Optimizing the high-frequency features of different scales and the low-frequency features of different scales to enhance the spatial texture and spectral detail information of the satellite remote sensing image.

[0027] In a possible implementation, the method further includes:

[0028] Performing feature fusion on the optimized high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features by minimizing a preset loss function to obtain a reconstructed target pan-sharpened image.

[0029] The present invention also provides a pan-sharpening device for satellite remote sensing images, including the following modules:

[0030] An acquisition module, configured to acquire a panchromatic-mode satellite remote sensing image and a multi-spectral-mode satellite remote sensing image to be processed;

[0031] An image processing module, configured to perform upsampling processing on the multi-spectral-mode satellite remote sensing image until reaching the target output size of the target pan-sharpened image, to obtain a first satellite remote sensing image of the multi-spectral mode with the target output size;

[0032] A feature extraction module, configured to perform multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image;

[0033] A feature processing module, configured to perform frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image, to obtain high-frequency features of different scales and low-frequency features of different scales;

[0034] An image reconstruction module, configured to perform feature fusion on the high-frequency features and low-frequency features of the same scale, and perform image reconstruction based on the fused features to obtain a reconstructed target pan-sharpened image.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for panchromatic sharpening of satellite remote sensing images as described in any one of the above is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for panchromatic sharpening of satellite remote sensing images as described in any one of the above is implemented.

[0037] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for panchromatic sharpening of a satellite remote sensing image as described in any one of the above is implemented.

[0038] The present invention provides a method, device, electronic device and medium for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the existing technology that the panchromatic sharpening and reconstruction of remote sensing images lacks high-frequency texture, does not conform to the perception of the human eye, and has an overall inconsistent look and feel, this solution dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner. At the same time, the image is upsampled to finely learn the high-frequency details of each scale. Finally, the image is reconstructed through a preset loss function to improve the detail richness and visual coordination of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 It is one of the schematic flowcharts of the panchromatic sharpening method for satellite remote sensing images provided by the present invention.

[0041] Figure 2 It is the second schematic flowchart of the panchromatic sharpening method for satellite remote sensing images provided by the present invention.

[0042] Figure 3 It is the schematic diagram of the panchromatic sharpening network structure provided by the present invention.

[0043] Figure 4 It is the schematic diagram of the multi-scale receptive field encoder structure provided by the present invention.

[0044] Figure 5 It is the schematic diagram of the frequency selective enhancement module provided by the present invention.

[0045] Figure 6 It is the schematic diagram of the HiFE, LoFE, and DPM module structures provided by the present invention.

[0046] Figure 7 It is the schematic diagram of the visual comparison on the Worldview-3 satellite provided by the present invention.

[0047] Figure 8 It is the schematic diagram of the structure of the panchromatic sharpening device for satellite remote sensing images provided by the present invention.

[0048] Figure 9 It is the schematic diagram of the structure of the electronic device provided by the present invention. Detailed implementation manners

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0050] For ease of understanding of the embodiments of the present invention, the following will further explain and illustrate with specific embodiments with reference to the accompanying drawings. The embodiments do not constitute a limitation to the embodiments of the present invention.

[0051] Figure 1 It is one of the schematic flowcharts of the panchromatic sharpening method for satellite remote sensing images provided by the present invention. As Figure 1 shown, the method includes the following:

[0052] S11. Obtain the satellite remote sensing image in the panchromatic modality and the satellite remote sensing image in the multi-spectral modality to be processed.

[0053] Panchromatic (PAN) images have high spatial resolution but low spectral resolution; multi-spectral (MS) images have high spectral resolution but limited spatial resolution. These two types of satellite remote sensing images can be obtained through remote sensing data and application service platforms, commercial satellite data providers, international data platforms, high-resolution satellite data, or geospatial data clouds.

[0054] S12. Upsample the satellite remote sensing image in the multi-spectral modality until it reaches the target output size of the target pan-sharpened image, obtaining the first satellite remote sensing image in the multi-spectral modality with the target output size.

[0055] The purpose of upsampling is to increase the spatial resolution of the satellite remote sensing image in the multi-spectral modality to the same size as the target pan-sharpened image, obtaining the first satellite remote sensing image. The upsampling methods used include, but are not limited to, bilinear interpolation, bicubic interpolation, etc. Through upsampling, the multi-spectral image can match the panchromatic image in terms of spatial resolution, providing a basis for subsequent feature extraction and fusion.

[0056] S13. Extract multi-scale features from the satellite remote sensing image in the panchromatic modality, obtaining different-scale features of the satellite remote sensing image in the panchromatic modality.

[0057] The purpose of multi-scale feature extraction is to obtain spatial information at different scales from the panchromatic image. For example, convolutional neural networks (CNNs) or Transformer architectures can be used to extract multi-scale features. These features can reflect the detailed information in the image, providing rich spatial details for subsequent image fusion.

[0058] S14. Perform frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image in the panchromatic modality with the same scale as the first satellite remote sensing image, obtaining high-frequency features at different scales and low-frequency features at different scales.

[0059] The purpose of frequency-selective enhancement processing is to separate high-frequency features and low-frequency features from the image features. High-frequency features usually contain the edge and detail information of the image, while low-frequency features reflect the overall structure of the image. By enhancing the high-frequency features, the details of the image can be better retained, and by retaining the low-frequency features, the overall structure of the image is ensured not to be lost.

[0060] Specifically, an upsampled image feature of a first satellite remote sensing image is obtained; dynamic frequency decoupling is performed on the upsampled image feature corresponding to the first satellite remote sensing image and the image feature of a satellite remote sensing image in a panchromatic mode having the same scale as the first satellite remote sensing image to obtain high-frequency information and low-frequency information of different scales; and frequency selective enhancement processing is performed on the high-frequency information and low-frequency information of different scales to obtain high-frequency features and low-frequency features of different scales.

[0061] S15, performing feature fusion on high-frequency features and low-frequency features of the same scale, and reconstructing an image based on the fused features to obtain a reconstructed target full-color sharpened image.

[0062] The high-frequency features and low-frequency features of the same scale are fused. Feature fusion is to organically combine high-frequency and low-frequency features of the same scale to generate richer image information. For example, weighted summation or cascade convolution can be used for fusion. The fused features are used to reconstruct the target panchromatic sharpened image. The resulting image has both the high spatial resolution of the panchromatic image and the spectral information of the multispectral image.

[0063] The present invention provides a method for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the panchromatic sharpening reconstruction of remote sensing images in the prior art, which lack high-frequency texture, do not conform to the perception of the human eye, and have an overall inconsistent look and feel, this method dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner, while upsampling the image to finely learn the high-frequency details of each scale, thereby improving the detail richness of the reconstructed image.

[0064] Figure 2 FIG. 2 is a flow chart of the method for pan-sharpening satellite remote sensing images provided by the present invention. Figure 2 As shown, the method includes the following:

[0065] In the embodiment of the present invention, pansharpening of satellite remote sensing images is achieved through a pansharpening network with frequency-targeted enhanced restoration of fine texture. The schematic diagram of the network structure is as Figure 3 shown. The backbone of the network is a progressive upsampling structure. The input multi-spectral modality satellite remote sensing image is upsampled twice to the output size of the reconstructed image. The input panchromatic modality satellite remote sensing image extracts multi-scale features through a multi-scale feature extractor. The features of the multi-spectral modality image after each upsampling and the features of the panchromatic modality image at the corresponding scale are jointly input into the Frequency Selective Augment Module (FSAM) to perform Dynamic Frequency Decouple (DFD), High Frequency Augment (HiFA), Low Frequency Augment (LoEF), Frequency Couple (FC), and Information Aggregation (IA) in sequence to obtain enhanced spatial texture and spectral details. The feature information output at the final size passes through the Fusion Block (FB) for fusion to obtain the reconstructed target pansharpened image.

[0066] S21. Obtain the satellite remote sensing image of the panchromatic modality and the satellite remote sensing image of the multi-spectral modality to be processed.

[0067] The panchromatic modality image (Panchromatic, PAN) has a high spatial resolution but a low spectral resolution; the multi-spectral modality image (Multi-Spectral Image, MS) has a high spectral resolution but a limited spatial resolution. These two modalities of satellite remote sensing images can be obtained through a remote sensing data and application service platform, commercial satellite data providers, international data platforms, high-resolution satellite data, or geospatial data clouds.

[0068] S22. Perform upsampling processing on the satellite remote sensing image of the multi-spectral modality until reaching the target output size of the target pansharpened image to obtain the first satellite remote sensing image of the multi-spectral modality with the target output size.

[0069] The purpose of upsampling is to increase the spatial resolution of the satellite remote sensing image in the multispectral modality to the same size as the target panchromatic sharpened image, obtaining the first satellite remote sensing image. The input multispectral image is gradually enlarged in size through two upsampling operations. After each upsampling, the resolution of the multispectral image will gradually increase to facilitate fusion with the high-resolution features of the panchromatic image. The upsampling methods used include, but are not limited to, bilinear interpolation, bicubic interpolation, etc. Through upsampling, the multispectral image can match the panchromatic image in terms of spatial resolution, providing a basis for subsequent feature extraction and fusion.

[0070] S23. Perform a convolution operation on the satellite remote sensing image in the panchromatic modality to extract the local features of the satellite remote sensing image in the panchromatic modality.

[0071] S24. Perform multiple pooling operations on the satellite remote sensing image in the panchromatic modality to extract the multi-scale features of the satellite remote sensing image in the panchromatic modality.

[0072] S25. Based on the local features and the multi-scale features, obtain the different-scale features of the satellite remote sensing image in the panchromatic modality.

[0073] The following is a unified description of S23 - S25:

[0074] The satellite remote sensing image in the panchromatic modality passes through a multi-scale feature extractor to extract feature information at different scales. This feature information can capture the texture and structural details of the image at different resolutions, providing rich information for subsequent fusion.

[0075] Convolution operation is a common method for extracting local features of an image. By sliding a convolution kernel over the panchromatic image, local features such as edges and textures of the image can be extracted. This process is usually implemented through a convolutional neural network.

[0076] Pooling operations are used to extract multi-scale features of an image. By reducing the spatial resolution of the image, important information is retained. Multiple pooling operations can generate feature maps at different scales for subsequent multi-scale feature fusion.

[0077] Combine the local features extracted by the convolution operation with the multi-scale features generated by the pooling operation to obtain a multi-scale feature representation of the panchromatic image. These features can better reflect the details and structure of the image.

[0078] In the embodiments of the present invention, a multi-scale feature extractor (Feature Extractor) is used to perform multi-scale feature extraction on the panchromatic modality satellite remote sensing image. The input of the multi-scale feature extractor is a panchromatic modality image with a size of 128. After performing convolution operations, multi-scale features with sizes of 128, 64, and 32 are obtained through two max pooling layers (MaxPool) respectively. The feature extractor is composed of a multi-scale receptive field encoder module (Multiple Feature Encoder), a convolution module, and a pooling layer. Remote sensing images have large continuous ground object blocks (such as sea level, valleys, highways, etc.), as well as small scattered small targets (such as buildings, vehicles, ships, etc.). Therefore, the feature extractor needs to have a multi-scale receptive field, that is, it can capture the features of large-scale targets and also capture the edges and textures of small-scale targets. The multi-scale receptive field encoder module uses convolution modules with convolution kernel sizes of 7, 5, 3, and 1 for feature extraction. Considering that large-scale convolution kernels will bring huge computational overhead, referring to the practice of VGG, three consecutive 3×3 convolutions are used instead of 7×7 convolution, and two consecutive 3×3 convolutions are used instead of 5×5 convolution. The module structure is as Figure 4 shown.

[0079] S26. Obtain the upsampled image features of the first satellite remote sensing image.

[0080] Process the upsampled multi-spectral image through a feature extraction network (such as a convolutional neural network or Transformer) to extract its feature information. These features will be used for subsequent frequency decoupling and fusion.

[0081] S27. Perform dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the panchromatic modality satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales.

[0082] S28. Perform frequency-selective enhancement processing on the high-frequency information of different scales and the low-frequency information of different scales respectively to obtain high-frequency features of different scales and low-frequency features of different scales.

[0083] The input features of the frequency-selective enhancement module go through N frequency decoupling and frequency fusion operations, and finally the frequency features are integrated by an information aggregation module (IA) to enrich the non-linear feature expression and then output. The network structure of FSAM is as Figure 5 shown. Let the module input feature be F. Frequency decoupling uses a learnable filter composed of a pooling operation and a convolution to dynamically separate high and low frequency information. After decoupling, low-frequency information FL (such as the overall structure) with a size half of F is obtained. Then FL is upsampled by a factor of two and dynamically filtered to have the same size as F, and subtracted from F to obtain high-frequency feature FH (such as details and textures).

[0084] FH and FL are respectively subjected to high-frequency enhancement (HiFA) and low-frequency enhancement (LoFA). Through the enhancement process, the details and contrast of the image can be further improved. The module structure is as shown in Figure 6 . For FH, a self-attention mechanism is adopted to utilize global features, fully explore the long-range correlation of FH, and strengthen it. The dual-path module (DPM) is used in both HiFA and LoFA to enhance features. DPM adopts a dual-branch convolution structure, and the input enters two different-depth convolution paths respectively. High-level semantic information such as texture is enhanced in the deep path, and low-level semantic information such as edges is enhanced in the shallow path.

[0085] FH and FL enter the frequency fusion module (Frequency Couple), and after convolution calculation, spatial attention (SA) and channel attention calculation (CA), and convolution calculation, the output is obtained. After N such processes, the features are further enriched in the non-linear expression of features and the high-frequency and low-frequency features are fused through the information aggregation module (IA) and then output. The IA module draws on the inverted bottleneck structure and the ConvFFN structure. The input first passes through a large-scale depthwise convolution (Depth-wise Conv, DW), which enables the module to have a large receptive field, helps restore the details of the image, and at the same time limits the increase in computational overhead. After using DW, the GELU activation function is adopted. When the GELU function is close to zero, the gradient is not zero, and the derivative function is smoother, which further retains details and aggregates information.

[0086] The fusion block (FB) consists of a 3×3 convolution, the LeakyReLu and Tanh activation functions, and gradually restores the feature channels to the number of channels of the fused image, ensuring the retention of detail features to the greatest extent.

[0087] S29. Feature fusion is performed on the optimized high-frequency and low-frequency features of the same scale, and image reconstruction is performed based on the fused features by minimizing a preset loss function to obtain the reconstructed target panchromatic sharpened image.

[0088] After fusing the high-frequency and low-frequency features, image reconstruction is performed through a deep learning network (such as Pan-Mamba or DUNet-HA). By minimizing the preset loss function, the quality of the reconstructed image is optimized, and finally the target panchromatic sharpened image is generated.

[0089] The preset loss function adopts a combined loss of L1 loss and perceptual loss to ensure the detail quality and overall coordination of the reconstructed image. The formula is expressed as follows:

[0090] (1)

[0091] Among them, \(L\) is the loss in the low-resolution stage, is the low-resolution loss, is the high-resolution loss, and is the perceptual loss. Three different coefficients \(\alpha\), \(\beta\), and \(\gamma\) are used to weigh these loss terms.

[0092] First The loss is used to ensure that the fused image with low resolution still has the same spectral distribution as the low-resolution multispectral image under the condition of spatial resolution degradation, and can be expressed as:

[0093] (2)

[0094] Among them, \(F\downarrow\) means that the fused image of \(128\times128\) is downsampled by 4 times to obtain a fused image of \(32\times32\), and then the L1 loss is calculated with the low-resolution multispectral image \(LRMS\) of \(32\times32\). Minimizing this loss helps to more accurately retain the spectral distribution. Intuitively, any fused image should also be as similar as possible to the original multispectral image after the spatial resolution degrades.

[0095] Secondly, use the loss to constrain the spectral information of the fused image to be consistent with the observed multispectral image, and the formula is:

[0096] (3)

[0097] In the image reconstruction task, the L1 loss helps to retain edges and details. The L1 loss does not overly penalize large pixel differences, which are often important high-frequency features in remote sensing images.

[0098] The perceptual loss used is the VGG Perceptual Loss. The difference between the perceptual loss and the L1 loss is that the L1 loss calculates the difference using image pixels, and the perceptual loss is calculated in the feature space. The formula for the VGG perceptual loss is as follows:

[0099] (4)

[0100] Among them, represents the VGG network, represents the feature map of the \(j\)-th layer of the network, represents the size of the feature map of the \(j\)-th layer, \(y\) represents the true \(HRMS\), Denote the fusion HRMS. This loss extracts features from the fused image and the ground truth image, and calculates the L2 loss. This loss conforms to human visual perception, helps to restore image details, and improves image consistency.

[0101] The pan-sharpening method for satellite remote sensing images provided by the present invention aims at the problem of insufficient high-frequency information restoration ability of existing methods, and designs a frequency-selective enhancement module. This module dynamically separates image features and enhances them specifically to strengthen the richness of high-frequency details in the fused image. At the same time, a progressive upsampling learning network is constructed to learn fine details at different scales of the picture. Aiming at the problems of insufficient richness of details and poor human visual perception of existing methods, a loss function combination of L1 loss and perceptual loss is designed, so that the reconstructed image has rich texture details and high visual consistency.

[0102] To verify the effectiveness of the proposed method, the embodiments of the present invention also compare its performance with several advanced pan-sharpening methods. Specifically, five traditional methods, namely IHS, GFPC, GS, Brovey, and SFIM, and five deep learning-based methods, namely Pannet (ICCV 2017), Srppnn (TGRS 2021), Gppnn (CVPR 2011), HyperTransforme (CVPR 2022), and SFINet++ (TPAMI 2024), are selected. The experimental results of the method proposed in the embodiments of the present invention on the Worldview-3 dataset are shown in Table 1. As can be seen from Table 1, the proposed method reaches the state-of-the-art performance level in the main indicators, which verifies the effectiveness of the proposed method.

[0103] Table 1 Performance on Worldview-3

[0104]

[0105] The qualitative performance on the Worldview-3 dataset is as Figure 7 shown, Figure 7 The lower part shows the mean square error between the fused image and the Ground Truth. The darker the color, the smaller the error value; conversely, the lighter the color, the larger the error. As can be seen from the figure, compared with other methods, the overall pixel color of the proposed method on the error map is darker, indicating that the result is very close to the GT and is better than other methods. In the enlarged area of the residual map, the proposed method is generally closer to dark blue. The error pixel colors in the upper left corner of the enlarged area are close to green and dark blue, while other methods have bright yellow pixel points in the upper left corner, involving more yellow, which shows that the proposed method has better spatial fidelity performance and high-frequency detail restoration performance. This visual comparison further proves that the proposed method achieves the state-of-the-art performance and is superior to existing pan-sharpening algorithms.

[0106] The present invention provides a method for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the existing technology that the panchromatic sharpening and reconstruction of remote sensing images lacks high-frequency texture, does not conform to the perception of the human eye, and has an overall inconsistent look and feel, this method dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner. At the same time, the image is upsampled to finely learn the high-frequency details of each scale. Finally, the image is reconstructed through a preset loss function to improve the detail richness and visual coordination of the reconstructed image.

[0107] The full-color sharpening device for satellite remote sensing images provided by the present invention is described below. The full-color sharpening device for satellite remote sensing images described below and the full-color sharpening method for satellite remote sensing images described above can be referred to each other.

[0108] Figure 8 The structure diagram of the panchromatic sharpening device for satellite remote sensing images provided by the present invention specifically comprises:

[0109] The acquisition module 801 is used to acquire the panchromatic satellite remote sensing image and the multispectral satellite remote sensing image to be processed. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0110] The image processing module 802 is used to upsample the satellite remote sensing image in the multispectral mode until the target output size of the target pan-sharpened image is reached, and obtain the first satellite remote sensing image in the multispectral mode of the target output size. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0111] The feature extraction module 803 is used to extract multi-scale features from the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0112] A feature processing module 804 is configured to perform frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image in the panchromatic modality having the same scale as the first satellite remote sensing image, so as to obtain high-frequency features of different scales and low-frequency features of different scales. For detailed description, refer to the relevant description corresponding to the above method embodiment, which will not be elaborated here.

[0113] An image reconstruction module 805 is configured to perform feature fusion on the high-frequency features and low-frequency features having the same scale, and perform image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. For detailed description, refer to the relevant description corresponding to the above method embodiment, which will not be elaborated here.

[0114] Figure 9 An example of a schematic physical structure diagram of an electronic device is shown as Figure 9 shown. The electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communication interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 can call the logical instructions in the memory 930 to execute the panchromatic sharpening method for satellite remote sensing images. The method includes obtaining a satellite remote sensing image in the panchromatic modality and a satellite remote sensing image in the multispectral modality to be processed; performing upsampling processing on the satellite remote sensing image in the multispectral modality until reaching the target output size of the target panchromatic sharpened image, so as to obtain a first satellite remote sensing image in the multispectral modality with the target output size; performing multi-scale feature extraction on the satellite remote sensing image in the panchromatic modality to obtain different scale features of the satellite remote sensing image in the panchromatic modality; performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image in the panchromatic modality having the same scale as the first satellite remote sensing image, so as to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on the high-frequency features and low-frequency features having the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0115] In addition, when the logical instructions in the above-mentioned memory 930 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the panchromatic sharpening method of satellite remote sensing images provided by the above-mentioned various methods. The method includes: acquiring a panchromatic-mode satellite remote sensing image and a multi-spectral-mode satellite remote sensing image to be processed; performing upsampling processing on the multi-spectral-mode satellite remote sensing image until the target output size of the target panchromatic sharpened image is reached, to obtain a first satellite remote sensing image of the multi-spectral mode with the target output size; performing multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image; performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency features and low-frequency features of different scales; fusing the high-frequency features and low-frequency features with the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements a pan-sharpening method for satellite remote sensing images provided by the above-mentioned various methods. The method includes: acquiring a panchromatic-mode satellite remote sensing image and a multi-spectral-mode satellite remote sensing image to be processed; performing upsampling processing on the multi-spectral-mode satellite remote sensing image until reaching the target output size of the target pan-sharpened image, to obtain a first satellite remote sensing image of the multi-spectral mode with the target output size; performing multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image; performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image, to obtain high-frequency features of different scales and low-frequency features of different scales; fusing the high-frequency features and low-frequency features with the same scale, and performing image reconstruction based on the fused features, to obtain a reconstructed target pan-sharpened image.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0119] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0120] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panchromatic sharpening method for satellite remote sensing images, characterized in that, Including: Obtain a panchromatic-mode satellite remote sensing image and a multispectral-mode satellite remote sensing image to be processed; Perform upsampling processing on the multispectral-mode satellite remote sensing image until reaching the target output size of the target panchromatic sharpened image, to obtain a first satellite remote sensing image of the multispectral mode with the target output size; Perform multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image; The performing multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image includes: Perform convolution operation on the panchromatic-mode satellite remote sensing image to extract local features of the panchromatic-mode satellite remote sensing image; Perform multiple pooling operations on the panchromatic-mode satellite remote sensing image to extract multi-scale features of the panchromatic-mode satellite remote sensing image; Based on the local features and the multi-scale features, obtain different-scale features of the panchromatic-mode satellite remote sensing image; Perform frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; The performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales includes: Obtain the upsampled image features of the first satellite remote sensing image; Perform dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales; Perform frequency-selective enhancement processing on the high-frequency information of different scales and the low-frequency information of different scales respectively to obtain high-frequency features of different scales and low-frequency features of different scales; Fuse the high-frequency features and low-frequency features with the same scale, and perform image reconstruction based on the fused features to obtain the reconstructed target panchromatic sharpened image.

2. The method according to claim 1, wherein After the performing dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales, it includes: Based on a preset frequency fusion strategy, perform frequency fusion on the high-frequency information and low-frequency information with the same scale to obtain frequency information features of different scales after fusion.

3. The method according to claim 2, wherein The performing frequency-selective enhancement processing on the high-frequency information of different scales and the low-frequency information of different scales respectively to obtain high-frequency features of different scales and low-frequency features of different scales includes: Perform frequency-selective enhancement processing on the frequency information features of different scales after fusion to obtain high-frequency features of different scales and low-frequency features of different scales.

4. The method according to claim 3, wherein The method further includes: Optimize the high-frequency features and low-frequency features of different scales to enhance the spatial texture and spectral detail information of the satellite remote sensing image.

5. The method according to claim 4, wherein Fuse the high-frequency features and low-frequency features of the same scale, and perform image reconstruction based on the fused features to obtain the reconstructed target pan-sharpened image, including: Fuse the optimized high-frequency features and low-frequency features of the same scale, and perform image reconstruction by minimizing a preset loss function based on the fused features to obtain the reconstructed target pan-sharpened image.

6. A panchromatic sharpening device for satellite remote sensing images, characterized in that, Including: An acquisition module, configured to acquire a panchromatic-mode satellite remote sensing image and a multi-spectral-mode satellite remote sensing image to be processed; An image processing module, configured to perform upsampling processing on the multi-spectral-mode satellite remote sensing image until reaching the target output size of the target pan-sharpened image, to obtain a first satellite remote sensing image of the multi-spectral mode with the target output size; A feature extraction module, configured to perform multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image; The performing multi-scale feature extraction on the panchromatic-mode satellite remote sensing image to obtain different-scale features of the panchromatic-mode satellite remote sensing image includes: performing a convolution operation on the panchromatic-mode satellite remote sensing image to extract local features of the panchromatic-mode satellite remote sensing image; performing multiple pooling processes on the panchromatic-mode satellite remote sensing image to extract multi-scale features of the panchromatic-mode satellite remote sensing image; based on the local features and the multi-scale features, obtaining different-scale features of the panchromatic-mode satellite remote sensing image; A feature processing module, configured to perform frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency features and low-frequency features of different scales; the performing frequency-selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency features and low-frequency features of different scales includes: acquiring the upsampled image features of the first satellite remote sensing image; Performing dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the panchromatic-mode satellite remote sensing image with the same scale as the first satellite remote sensing image to obtain high-frequency information and low-frequency information of different scales; performing frequency-selective enhancement processing on the high-frequency information and the low-frequency information of different scales respectively to obtain high-frequency features and low-frequency features of different scales; An image reconstruction module, configured to fuse the high-frequency features and low-frequency features of the same scale, and perform image reconstruction based on the fused features to obtain the reconstructed target pan-sharpened image.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the pan-sharpening method of the satellite remote sensing image according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the panchromatic sharpening method for satellite remote sensing images according to any one of claims 1 to 5.

Images