Satellite remote sensing image panchromatic sharpening method and device, electronic equipment and medium

Abstract

The embodiment of the invention relates to the technical field of remote sensing image processing, and provides a panchromatic sharpening method and device for a satellite remote sensing image, electronic equipment and a medium, and the method comprises the steps: obtaining a to-be-processed satellite remote sensing image in a panchromatic mode and a multispectral mode; performing up-sampling processing on the multispectral modal image until the target output size of the target panchromatic sharpening image is reached, and obtaining a multispectral modal first satellite remote sensing image; performing multi-scale feature extraction on the panchromatic modal image to obtain different scale features of the panchromatic modal satellite remote sensing image; performing frequency selectivity enhancement processing on image features of the first satellite remote sensing image and the panchromatic modal image with the same scale as the first satellite remote sensing image to obtain high-frequency features and low-frequency features with different scales; and carrying out feature fusion on the high and low frequency features with the same scale, and carrying out image reconstruction to obtain a target panchromatic sharpening image. Therefore, the detail richness and the visual coordination of the reconstructed image can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a method, device, electronic equipment and medium for panchromatic sharpening of satellite remote sensing images. Background Art

[0002] Panchromatic sharpening is one of the key processing links of satellite images, which has great application value and broad prospects. Panchromatic sharpening technology complements and fuses panchromatic and multispectral remote sensing images, overcomes the shortcomings of insufficient information from single-modal remote sensing images, and maximizes the advantages of each modality, which plays an important role in downstream remote sensing tasks such as change detection, recognition, and detection.

[0003] Under constraints, full-color images have high spatial resolution, but can only provide single-channel band information. Multispectral images have rich spectral information, but their spatial resolution is poor. In the fusion process, it is necessary to combine the advantages of different modalities, fully extract the features of each modality, fuse complementary features, and remove redundant features. Existing research is based on deep learning to model the mapping relationship between the two modalities. Modal features are proposed through convolutional neural networks, and Transformer is used for long-distance feature interaction to model the correlation of cross-modal features. Some researchers also convert the image to the frequency domain first, and then model the spectral relationship through convolutional neural networks. In the reconstruction process, the loss function of these methods uses L1 loss or L2 loss.

[0004] A common approach is to use the global modeling capabilities of the Transformer mechanism to interact with cross-modal features or use transformation tools to convert images to the frequency domain, and combine the frequency domain and spatial domain to learn spectral distribution features and spatial distribution features. However, in the process of neural network learning, the network easily learns low-frequency features and tends to ignore high-frequency features. Some high-frequency information will be smoothed by the network or lost as an outlier. The above methods do not selectively enhance the high-frequency and low-frequency parts of image features differently, which will result in the lack of high-frequency texture in the reconstructed image.

[0005] When using L2 loss, the network tends to smooth out points with larger pixel values, which are often high-frequency details in remote sensing images. When only L1 loss and L2 loss are used as reconstruction losses, the reconstructed image is clear but does not conform to human eye perception and the overall look and feel is not harmonious. Summary of the invention

[0006] The present invention provides a method, device, electronic device and medium for panchromatic sharpening of satellite remote sensing images, which are used to solve the defects of panchromatic sharpening and reconstructing remote sensing images in the prior art, such as lack of high-frequency texture, incompatibility with human eye perception and overall inharmonious appearance, so as to improve the detail richness and visual coordination of fused images.

[0007] The present invention provides a method for panchromatic sharpening of satellite remote sensing images, comprising: Acquire a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; Upsampling the multispectral satellite remote sensing image until a target output size of a target pan-sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; Performing multi-scale feature extraction on the satellite remote sensing image in the panchromatic modality to obtain different scale features of the satellite remote sensing image in the panchromatic modality; Performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image having the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; The high-frequency features and low-frequency features of the same scale are fused, and the image is reconstructed based on the fused features to obtain a reconstructed full-color sharpened image of the target.

[0008] In a possible implementation, the method further includes: Performing a convolution operation on the panchromatic satellite remote sensing image to extract local features of the panchromatic satellite remote sensing image; Performing multiple pooling processes on the panchromatic satellite remote sensing image to extract multi-scale features of the panchromatic satellite remote sensing image; Based on the local features and the multi-scale features, different scale features of the panchromatic satellite remote sensing image are obtained.

[0009] In a possible implementation, the method further includes: Acquire upsampled image features of the first satellite remote sensing image; Performing dynamic frequency decoupling on upsampled image features corresponding to the first satellite remote sensing image and image features of a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales; The high-frequency information of different scales and the low-frequency information of different scales are respectively subjected to frequency selective enhancement processing to obtain high-frequency features of different scales and low-frequency features of different scales.

[0010] In a possible implementation, the method further includes: Based on the preset frequency fusion strategy, the high-frequency information and low-frequency information of the same scale are frequency fused to obtain the fused frequency information features of different scales.

[0011] In a possible implementation, the method further includes: The frequency information features of different scales are subjected to frequency selective enhancement processing to obtain high-frequency features of different scales and low-frequency features of different scales.

[0012] In a possible implementation, the method further includes: The high-frequency features of different scales and the low-frequency features of different scales are optimized to enhance the spatial texture and spectral detail information of the satellite remote sensing image.

[0013] In a possible implementation, the method further includes: The optimized high-frequency features and low-frequency features of the same scale are fused, and the image is reconstructed based on the fused features by minimizing the preset loss function to obtain the reconstructed target full-color sharpened image.

[0014] The present invention also provides a device for panchromatic sharpening of satellite remote sensing images, comprising the following modules: An acquisition module, used for acquiring a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; An image processing module is used to perform upsampling processing on the satellite remote sensing image of the multispectral modality until a target output size of the target pan-sharpened image is reached, and obtain a first satellite remote sensing image of the multispectral modality of the target output size; A feature extraction module is used to perform multi-scale feature extraction on the satellite remote sensing image in the panchromatic mode to obtain different scale features of the satellite remote sensing image in the panchromatic mode; a feature processing module, configured to perform frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image having the same scale as the first satellite remote sensing image, so as to obtain high-frequency features of different scales and low-frequency features of different scales; The image reconstruction module is used to fuse the high-frequency features and low-frequency features of the same scale, and reconstruct the image based on the fused features to obtain a reconstructed target full-color sharpened image.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for panchromatic sharpening of satellite remote sensing images as described in any one of the above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for panchromatic sharpening of satellite remote sensing images as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for panchromatic sharpening of a satellite remote sensing image as described in any one of the above is implemented.

[0018] The present invention provides a method, device, electronic device and medium for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the existing technology that the panchromatic sharpening and reconstruction of remote sensing images lacks high-frequency texture, does not conform to the perception of the human eye, and has an overall inconsistent look and feel, this solution dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner. At the same time, the image is upsampled to finely learn the high-frequency details of each scale. Finally, the image is reconstructed through a preset loss function to improve the detail richness and visual coordination of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is one of the flow charts of the panchromatic sharpening method for satellite remote sensing images provided by the present invention.

[0021] Figure 2 This is the second flow chart of the method for panchromatic sharpening of satellite remote sensing images provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the full color sharpening network structure provided by the present invention.

[0023] Figure 4 It is a schematic diagram of the structure of a multi-scale receptive field encoder provided by the present invention.

[0024] Figure 5 It is a schematic diagram of the frequency selective enhancement module provided by the present invention.

[0025] Figure 6 It is a schematic diagram of the HiFE, LoFE and DPM modules provided by the present invention.

[0026] Figure 7 It is a visualized comparison diagram on the Worldview-3 satellite provided by the present invention.

[0027] Figure 8 The present invention is a schematic structural diagram of a device for panchromatic sharpening of satellite remote sensing images.

[0028] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] To facilitate understanding of the embodiments of the present invention, specific embodiments will be further explained below in conjunction with the accompanying drawings. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0031] Figure 1 This is one of the flow charts of the method for panchromatic sharpening of satellite remote sensing images provided by the present invention, such as Figure 1 As shown, the method includes the following: S11. Obtain a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed.

[0032] Panchromatic (PAN) images have high spatial resolution but low spectral resolution, while multispectral (MS) images have high spectral resolution but limited spatial resolution. Satellite remote sensing images of these two modalities can be obtained through remote sensing data and application service platforms, commercial satellite data providers, international data platforms, high-resolution satellite data or geospatial data clouds.

[0033] S12. Upsampling the satellite remote sensing image in the multispectral modality until a target output size of the target pan-sharpened image is reached, thereby obtaining a first satellite remote sensing image in the multispectral modality in the target output size.

[0034] The purpose of upsampling is to increase the spatial resolution of the multispectral satellite remote sensing image to the same size as the target panchromatic sharpened image, and obtain the first satellite remote sensing image. The upsampling methods used include but are not limited to bilinear interpolation, bicubic interpolation, etc. Through upsampling, the multispectral image can match the panchromatic image in spatial resolution, providing a basis for subsequent feature extraction and fusion.

[0035] S13. Perform multi-scale feature extraction on the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image.

[0036] Multi-scale feature extraction aims to obtain spatial information at different scales from full-color images. For example, convolutional neural networks (CNN) or Transformer architectures can be used to extract multi-scale features. These features can reflect the detailed information in the image and provide rich spatial details for subsequent image fusion.

[0037] S14. Perform frequency selective enhancement processing on image features of the first satellite remote sensing image and a satellite remote sensing image of panchromatic mode having the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales.

[0038] The purpose of frequency selective enhancement is to separate high-frequency features and low-frequency features from image features. High-frequency features usually contain edge and detail information of the image, while low-frequency features reflect the overall structure of the image. By enhancing high-frequency features, the details of the image can be better preserved, while by retaining low-frequency features, the overall structure of the image is not lost.

[0039] Specifically, an upsampled image feature of a first satellite remote sensing image is obtained; dynamic frequency decoupling is performed on the upsampled image feature corresponding to the first satellite remote sensing image and the image feature of a satellite remote sensing image in a panchromatic mode having the same scale as the first satellite remote sensing image to obtain high-frequency information and low-frequency information of different scales; and frequency selective enhancement processing is performed on the high-frequency information and low-frequency information of different scales to obtain high-frequency features and low-frequency features of different scales.

[0040] S15, performing feature fusion on high-frequency features and low-frequency features of the same scale, and reconstructing an image based on the fused features to obtain a reconstructed target full-color sharpened image.

[0041] The high-frequency features and low-frequency features of the same scale are fused. Feature fusion is to organically combine high-frequency and low-frequency features of the same scale to generate richer image information. For example, weighted summation or cascade convolution can be used for fusion. The fused features are used to reconstruct the target panchromatic sharpened image. The resulting image has both the high spatial resolution of the panchromatic image and the spectral information of the multispectral image.

[0042] The present invention provides a method for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the panchromatic sharpening reconstruction of remote sensing images in the prior art, which lack high-frequency texture, do not conform to the perception of the human eye, and have an overall inconsistent look and feel, this method dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner, while upsampling the image to finely learn the high-frequency details of each scale, thereby improving the detail richness of the reconstructed image.

[0043] Figure 2 FIG. 2 is a flow chart of the method for pan-sharpening satellite remote sensing images provided by the present invention. Figure 2 As shown, the method includes the following: The embodiment of the present invention realizes the pan-sharpening of satellite remote sensing images through a pan-sharpening network with frequency-targeted enhancement and restoration of fine textures. The schematic diagram of the network structure is shown in FIG. Figure 3As shown in the figure, the backbone of the network is a progressive upsampling structure. The input multispectral satellite remote sensing image is upsampled twice to the reconstructed image output size. The input panchromatic satellite remote sensing image is extracted with multiscale features by a multiscale feature extractor. The features of the multispectral image after each upsampling are input into the frequency selective augmentation module (FSAM) together with the panchromatic image features of the corresponding scale, and then the dynamic frequency decoupling (DFD), high frequency augmentation (HiFA), low frequency augmentation (LoEF), frequency fusion (FC), and information aggregation (IA) are performed in sequence to obtain the enhanced spatial texture and spectral details. The feature information of the final size output is fused through the fusion module (FB) to obtain the reconstructed target panchromatic sharpened image.

[0044] S21. Obtain a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed.

[0045] Panchromatic (PAN) images have high spatial resolution but low spectral resolution, while multispectral (MS) images have high spectral resolution but limited spatial resolution. Satellite remote sensing images of these two modalities can be obtained through remote sensing data and application service platforms, commercial satellite data providers, international data platforms, high-resolution satellite data or geospatial data clouds.

[0046] S22. Up-sampling the satellite remote sensing image in the multispectral modality until a target output size of the target pan-sharpened image is reached, thereby obtaining a first satellite remote sensing image in the multispectral modality in the target output size.

[0047] The purpose of upsampling is to increase the spatial resolution of the satellite remote sensing image of the multispectral modality to the same size as the target panchromatic sharpened image, and obtain the first satellite remote sensing image. The input multispectral image is gradually enlarged in size through two upsampling operations. After each upsampling, the resolution of the multispectral image will gradually increase so that it can be fused with the high-resolution features of the panchromatic image. The upsampling methods used include but are not limited to bilinear interpolation, bicubic interpolation, etc. Through upsampling, the multispectral image can match the panchromatic image in spatial resolution, providing a basis for subsequent feature extraction and fusion.

[0048] S23. Perform a convolution operation on the panchromatic satellite remote sensing image to extract local features of the panchromatic satellite remote sensing image.

[0049] S24. Perform multiple pooling processes on the panchromatic satellite remote sensing image to extract multi-scale features of the panchromatic satellite remote sensing image.

[0050] S25. Based on the local features and the multi-scale features, different scale features of the satellite remote sensing image in the panchromatic mode are obtained.

[0051] The following is a unified description of S23-S25: The panchromatic satellite remote sensing image is extracted through a multi-scale feature extractor to extract feature information at different scales. These feature information can capture the texture and structural details of the image at different resolutions, providing rich information for subsequent fusion.

[0052] Convolution operation is a common method for extracting local features of an image. By sliding the convolution kernel on a full-color image, local features such as the edge and texture of the image can be extracted. This process is usually implemented through a convolutional neural network.

[0053] The pooling operation is used to extract multi-scale features of an image and retain important information by reducing the spatial resolution of the image. Multiple pooling operations can generate feature maps of different scales for subsequent multi-scale feature fusion.

[0054] The local features extracted by the convolution operation are combined with the multi-scale features generated by the pooling operation to obtain a multi-scale feature representation of the full-color image. These features can better reflect the details and structure of the image.

[0055] The embodiment of the present invention uses a multi-scale feature extractor to extract multi-scale features from a satellite remote sensing image in full-color mode. The input of the multi-scale feature extractor is a full-color modality image of size 128. After the convolution operation, multi-scale features of sizes 128, 64, and 32 are obtained respectively through two maximum pooling layers (MaxPool). The feature extractor is composed of a multi-scale receptive field encoder module (Multiple Feature Encoder), a convolution module, and a pooling layer. Remote sensing images have large areas of continuous land objects (sea level, valleys, roads, etc.), as well as small pieces of scattered small targets (buildings, vehicles, ships, etc.). Therefore, the feature extractor is required to have a multi-scale receptive field, that is, it can capture the features of large-scale targets and the edges and textures of small-scale targets. The multi-scale receptive field encoder module uses convolution modules with kernel sizes of 7, 5, 3, and 1 for feature extraction. Considering that large-scale convolution kernels will bring huge computational overhead, referring to the VGG approach, three consecutive 3×3 convolutions are used instead of 7×7 convolutions, and two consecutive 3×3 convolutions are used instead of 5×5 convolutions. The module structure is as follows: Figure 4 shown.

[0056] S26. Obtain upsampled image features of the first satellite remote sensing image.

[0057] The upsampled multispectral image is processed through a feature extraction network (such as a convolutional neural network or Transformer) to extract its feature information. These features will be used for subsequent frequency decoupling and fusion.

[0058] S27. Perform dynamic frequency decoupling on the upsampled image features corresponding to the first satellite remote sensing image and the image features of the satellite remote sensing image in the panchromatic mode with the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales.

[0059] S28, performing frequency selective enhancement processing on the high-frequency information of different scales and the low-frequency information of different scales respectively, to obtain high-frequency features of different scales and low-frequency features of different scales.

[0060] The input features of the frequency selective enhancement module undergo N frequency decoupling and frequency fusion operations, and finally the frequency features are integrated through the information aggregation module (IA) to enrich the nonlinear feature expression before output. The network structure of FSAM is as follows: Figure 5 As shown in Figure 2. Let the module input feature be F. Frequency decoupling uses a learnable filter composed of pooling operations and convolutions to dynamically separate high- and low-frequency information. After decoupling, low-frequency information FL (such as the overall structure) with a size half of F is obtained. Then FL is double-upsampled and dynamically filtered to the same size as F, and subtracted from F to obtain high-frequency features FH (such as details and textures).

[0061] FH and FL perform high frequency enhancement (HiFA) and low frequency enhancement (LoFA) respectively. Through the enhancement processing, the details and contrast of the image can be further improved. The module structure is as follows Figure 6 As shown in the figure. For FH, a self-attention mechanism is used to fully explore the long-range correlation of FH and strengthen it by using global features. Dual Path Module (DPM) is used in both HiFA and LoFA to strengthen the features. DPM adopts a dual convolution structure, and the input enters two convolution paths of different depths respectively. High-level semantic information such as texture is strengthened in the deep path, and low-level semantic information such as edge is strengthened in the shallow path.

[0062] FH and FL enter the frequency fusion module (Frequency Couple), and are output after convolution calculation, spatial attention (SA) and channel attention calculation (CA), and convolution calculation. After N of the above processes, the features are further enriched by the information aggregation module (IA) for nonlinear expression, and high- and low-frequency features are fused before output. The IA module draws on the inverted bottleneck structure and ConvFFN structure. The input first passes through a large-scale depth-wise separable convolution (Depth-wise Conv, DW), which gives the module a larger receptive field, helps restore image details, and limits the increase in computational overhead. After using DW, the GELU activation function is used. When the GELU function is close to zero, the gradient is not 0, and the derivative function is smoother, which further retains details and aggregates information.

[0063] The fusion block (FB) consists of 3×3 convolution and LeakyReLu and Tanh activation functions, which gradually restores the feature channels to the number of channels of the fused image to ensure the retention of detail features to the greatest extent.

[0064] S29, performing feature fusion on the optimized high-frequency features and low-frequency features of the same scale, and reconstructing the image based on the fused features by minimizing a preset loss function to obtain a reconstructed target full-color sharpened image.

[0065] After fusing the high-frequency and low-frequency features, the image is reconstructed through a deep learning network (such as Pan-Mamba or DUNet-HA). By minimizing the preset loss function, the quality of the reconstructed image is optimized and the target panchromatic sharpened image is finally generated.

[0066] The preset loss function uses a combination of L1 loss and perceptual loss to ensure the detail quality and overall coordination of the reconstructed image. The formula is as follows: (1) Where L is the loss in the resolution reduction stage, is the low resolution loss, is the high resolution loss, is the perceptual loss. Three different coefficients α, β, and γ are used to weigh these loss terms.

[0067] first The loss is used to ensure that the low-resolution fused image still has the same spectral distribution as the low-resolution multispectral image when the spatial resolution is degraded, which can be expressed as: (2) Where F↓ represents the 128×128 fused image being downsampled by 4 times to obtain a 32×32 fused image, which is then compared with the 32×32 low-resolution multispectral image LRMS to calculate the L1 loss. Minimizing this loss helps preserve the spectral distribution more accurately. Intuitively, any fused image should be as similar as possible to the original multispectral image after the spatial resolution degradation.

[0068] Secondly, use The loss is used to constrain the spectral information of the fused image to be consistent with the observed multispectral image. The formula is: (3) In image reconstruction tasks, the L1 loss helps preserve edges and details, and the L1 loss does not overly penalize large pixel differences, which are often important high-frequency features in remote sensing images.

[0069] The perceptual loss uses VGG perceptual loss (VGG Perceptual Loss). The difference between perceptual loss and L1 loss is that L1 loss uses image pixels for difference, while the perceptual loss is calculated in feature space. The VGG perceptual loss calculation formula is as follows: (4) in, represents the VGG network, represents the feature map of the jth layer of the network, represents the size of the feature map of the jth layer, y represents the true HRMS, Represents fused HRMS. This loss extracts features from the fused image and the real image and calculates the L2 loss. This loss is consistent with human eye perception, helps to restore image details, and improves image coordination.

[0070] The method for panchromatic sharpening of satellite remote sensing images provided by the present invention aims at the problem that the existing methods have insufficient ability to restore high-frequency information, and designs a frequency selective enhancement module, which dynamically separates image features and performs targeted enhancement to enhance the richness of high-frequency details in the fused image. At the same time, a progressive upsampling learning network is constructed to learn the fine details of the image at different scales. In view of the problem that the existing methods have insufficient detail richness and poor human eye perception, a loss function combination of L1 loss and perceptual loss is designed, so that the reconstructed image has rich texture details and high visual coordination.

[0071] In order to verify the effectiveness of the proposed method, the embodiment of the present invention also compares its performance with several advanced full color sharpening methods. Specifically, five traditional methods IHS, GFPC, GS, Brovey, SFIM, and five deep learning-based methods Pannet (ICCV 2017), Srppnn (TGRS 2021), Gppnn (CVPR 2011), HyperTransforme (CVPR 2022) and SFINet++ (TPAMI 2024) were selected. The experimental results of the method proposed in the embodiment of the present invention on the Worldview-3 dataset are shown in Table 1. It can be seen from Table 1 that the method has achieved the most advanced performance level in all major indicators, which verifies the effectiveness of the method.

[0072] Table 1 Performance on Worldview-3

[0073] The qualitative performance on the Worldview-3 dataset is as follows Figure 7 As shown, Figure 7 The lower part shows the mean square error of the fused image and the ground truth. The darker the color, the smaller the error value. Conversely, the brighter the color, the larger the error. As can be seen from the figure, the overall pixel color of this method on the error map is darker than that of other methods, indicating that the result is very close to GT and better than other methods. In the enlarged area of ​​the residual map, this method is generally closer to dark blue, and the error pixel color in the upper left corner of the enlarged area is close to green and dark blue, while other methods have bright yellow pixels in the upper left corner, involving more yellow, which shows that this method has better spatial fidelity and high-frequency detail recovery performance. This visual comparison further proves that this method achieves state-of-the-art performance and outperforms existing full-color sharpening algorithms.

[0074] The present invention provides a method for panchromatic sharpening of satellite remote sensing images. The method comprises the following steps: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; performing upsampling processing on the multispectral satellite remote sensing image until a target output size of a target panchromatic sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; performing multiscale feature extraction on the panchromatic satellite remote sensing image to obtain features of different scales of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image. Compared with the defects of the existing technology that the panchromatic sharpening and reconstruction of remote sensing images lacks high-frequency texture, does not conform to the perception of the human eye, and has an overall inconsistent look and feel, this method dynamically separates the feature information of satellite remote sensing images in panchromatic and multispectral modes into high-frequency and low-frequency information, and enhances them in a targeted manner. At the same time, the image is upsampled to finely learn the high-frequency details of each scale. Finally, the image is reconstructed through a preset loss function to improve the detail richness and visual coordination of the reconstructed image.

[0075] The full-color sharpening device for satellite remote sensing images provided by the present invention is described below. The full-color sharpening device for satellite remote sensing images described below and the full-color sharpening method for satellite remote sensing images described above can be referred to each other.

[0076] Figure 8 The structure diagram of the panchromatic sharpening device for satellite remote sensing images provided by the present invention specifically comprises: The acquisition module 801 is used to acquire the panchromatic satellite remote sensing image and the multispectral satellite remote sensing image to be processed. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0077] The image processing module 802 is used to upsample the satellite remote sensing image in the multispectral mode until the target output size of the target pan-sharpened image is reached, and obtain the first satellite remote sensing image in the multispectral mode of the target output size. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0078] The feature extraction module 803 is used to extract multi-scale features from the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0079] The feature processing module 804 is used to perform frequency selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image of the panchromatic mode with the same scale as the first satellite remote sensing image, so as to obtain high-frequency features of different scales and low-frequency features of different scales. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0080] The image reconstruction module 805 is used to fuse the high-frequency features and low-frequency features of the same scale, and reconstruct the image based on the fused features to obtain a reconstructed target full-color sharpened image. For detailed description, please refer to the relevant description corresponding to the above method embodiment, which will not be repeated here.

[0081] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor (processor) 910 , a communication interface (Communications Interface) 920 , a memory (memory) 930 and a communication bus 940 , wherein the processor 910 , the communication interface 920 , and the memory 930 communicate with each other through the communication bus 940 . The processor 910 can call the logic instructions in the memory 930 to execute the panchromatic sharpening method of the satellite remote sensing image, which includes obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; upsampling the multispectral satellite remote sensing image until the target output size of the target panchromatic sharpened image is reached, and obtaining a first satellite remote sensing image of the multispectral mode of the target output size; performing multi-scale feature extraction on the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on the high-frequency features and low-frequency features of the same scale, and reconstructing the image based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0082] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0083] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the panchromatic sharpening method of the satellite remote sensing image provided by the above methods, and the method includes: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; upsampling the multispectral satellite remote sensing image until the target output size of the target panchromatic sharpened image is reached, and obtaining a first multispectral satellite remote sensing image of the target output size; performing multi-scale feature extraction on the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on the high-frequency features and low-frequency features of the same scale, and reconstructing the image based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0084] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the panchromatic sharpening method of the satellite remote sensing image provided by the above-mentioned methods, the method comprising: obtaining a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; upsampling the multispectral satellite remote sensing image until the target output size of the target panchromatic sharpened image is reached, and obtaining a first multispectral satellite remote sensing image of the target output size; performing multi-scale feature extraction on the panchromatic satellite remote sensing image to obtain different scale features of the panchromatic satellite remote sensing image; performing frequency selective enhancement processing on the image features of the first satellite remote sensing image and the panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; performing feature fusion on the high-frequency features and low-frequency features of the same scale, and performing image reconstruction based on the fused features to obtain a reconstructed target panchromatic sharpened image.

[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for panchromatic sharpening of satellite remote sensing images, characterized in that: include: Acquire a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; Upsampling the multispectral satellite remote sensing image until a target output size of a target pan-sharpened image is reached, thereby obtaining a first multispectral satellite remote sensing image of the target output size; Performing multi-scale feature extraction on the satellite remote sensing image in the panchromatic modality to obtain different scale features of the satellite remote sensing image in the panchromatic modality; Performing frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image having the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales; The high-frequency features and low-frequency features of the same scale are fused, and the image is reconstructed based on the fused features to obtain a reconstructed full-color sharpened image of the target.

2. The method according to claim 1, characterized in that The multi-scale feature extraction is performed on the satellite remote sensing image in the panchromatic mode to obtain different scale features of the satellite remote sensing image in the panchromatic mode, including: Performing a convolution operation on the panchromatic satellite remote sensing image to extract local features of the panchromatic satellite remote sensing image; Performing multiple pooling processes on the panchromatic satellite remote sensing image to extract multi-scale features of the panchromatic satellite remote sensing image; Based on the local features and the multi-scale features, different scale features of the panchromatic satellite remote sensing image are obtained.

3. The method according to claim 1 or 2, characterized in that: The step of performing frequency selective enhancement processing on the image features of the first satellite remote sensing image and the satellite remote sensing image of the panchromatic mode having the same scale as the first satellite remote sensing image to obtain high-frequency features of different scales and low-frequency features of different scales includes: Acquire upsampled image features of the first satellite remote sensing image; Performing dynamic frequency decoupling on upsampled image features corresponding to the first satellite remote sensing image and image features of a panchromatic satellite remote sensing image of the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales; The high-frequency information of different scales and the low-frequency information of different scales are respectively subjected to frequency selective enhancement processing to obtain high-frequency features of different scales and low-frequency features of different scales.

4. The method according to claim 3, characterized in that After dynamically frequency decoupling the upsampled image features corresponding to the first satellite remote sensing image and the image features of the satellite remote sensing image in the panchromatic mode having the same scale as the first satellite remote sensing image to obtain high-frequency information of different scales and low-frequency information of different scales, the method includes: Based on the preset frequency fusion strategy, the high-frequency information and low-frequency information of the same scale are frequency fused to obtain the fused frequency information features of different scales.

5. The method according to claim 4, characterized in that The performing frequency selective enhancement processing on the high frequency information of different scales and the low frequency information of different scales respectively to obtain high frequency features of different scales and low frequency features of different scales includes: The fused frequency information features of different scales are subjected to frequency selective enhancement processing to obtain high-frequency features of different scales and low-frequency features of different scales.

6. The method according to claim 5, characterized in that The method further comprises: The high-frequency features of different scales and the low-frequency features of different scales are optimized to enhance the spatial texture and spectral detail information of the satellite remote sensing image.

7. The method according to claim 6, characterized in that The method of fusing high-frequency features and low-frequency features of the same scale and reconstructing an image based on the fused features to obtain a reconstructed target full-color sharpened image includes: The optimized high-frequency features and low-frequency features of the same scale are fused, and the image is reconstructed based on the fused features by minimizing the preset loss function to obtain the reconstructed target full-color sharpened image.

8. A device for panchromatic sharpening of satellite remote sensing images, characterized in that: include: An acquisition module, used for acquiring a panchromatic satellite remote sensing image and a multispectral satellite remote sensing image to be processed; An image processing module is used to perform upsampling processing on the satellite remote sensing image of the multispectral modality until a target output size of the target pan-sharpened image is reached, and obtain a first satellite remote sensing image of the multispectral modality of the target output size; A feature extraction module is used to perform multi-scale feature extraction on the satellite remote sensing image in the panchromatic mode to obtain different scale features of the satellite remote sensing image in the panchromatic mode; a feature processing module, configured to perform frequency selective enhancement processing on image features of the first satellite remote sensing image and a panchromatic satellite remote sensing image having the same scale as the first satellite remote sensing image, so as to obtain high-frequency features of different scales and low-frequency features of different scales; The image reconstruction module is used to fuse the high-frequency features and low-frequency features of the same scale, and reconstruct the image based on the fused features to obtain a reconstructed target full-color sharpened image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for pan-sharpening of satellite remote sensing images as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for pan-sharpening a satellite remote sensing image as claimed in any one of claims 1 to 7 is implemented.

Images