Frequency-space domain fusion panchromatic sharpening method and system based on DCT (Discrete Cosine Transform) and storage medium

Abstract

The invention discloses a frequency-spatial domain fusion panchromatic sharpening method and system based on DCT (Discrete Cosine Transform) and a storage medium, relates to the field of image processing, and solves the technical problems that lossless modal alignment of a panchromatic image and a multispectral image is difficult to realize and the multispectral image which not only keeps spectral fidelity but also has high spatial resolution is difficult to generate. According to the method, the dual-domain feature maps are extracted and spliced to realize modal alignment of the dual-domain feature maps, and then feature enhancement processing is performed through a channel attention mechanism and multi-scale fusion is performed to obtain a first fusion feature map; performing feature enhancement processing on a frequency domain feature map corresponding to the first fusion feature map and the first fusion feature map through a channel attention mechanism, and performing multi-scale fusion to obtain a second fusion feature map; performing residual connection on the second fusion feature map and the first fusion feature map to obtain a high-resolution multispectral image; according to the method, the spectral fidelity and the cross-scene stability can be considered while the spatial details are improved.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a frequency-spatial domain fusion pancolor sharpening method, system, and storage medium based on DCT. Background Technology

[0002] A representative wavelet multi-resolution fusion panchromatic sharpening process first performs a color space transformation on the original multispectral image to separate the luminance component, and then performs multi-scale wavelet decomposition on the high-resolution panchromatic image and the separated luminance component to obtain low-frequency approximation terms and several high-frequency detail sub-bands. Subsequently, according to a preset fusion rule, the high-frequency detail coefficients of the high-resolution panchromatic image are injected into or replaced in the corresponding scale sub-bands of the multispectral image. Finally, the high-resolution multispectral image is reconstructed through inverse wavelet transform and inverse color transform.

[0003] This method improves spatial resolution and preserves overall color to some extent; however, this type of method still has several inherent limitations that affect the fusion quality: 1. The rules used for subband fusion are mostly empirical designs or based on simple statistics. These static rules cannot be adaptively adjusted for different land cover types, texture complexity, or sensor response differences. Therefore, artifacts or over-sharpening are likely to occur in areas with complex textures, while in flat or noisy areas, they may not be able to effectively recover true details. 2. The wavelet decomposition and reconstruction process does not provide accurate compensation for the essential differences between high-resolution panchromatic images and multispectral images in terms of modes, such as resolution and spectral response, as well as the possible small registration errors. This results in the high-frequency information of the high-resolution panchromatic image not being accurately mapped to the correct spectral channels and spatial positions of the multispectral image, thus causing spectral shift or color distortion. 3. Although wavelet transform essentially provides separation of frequency components, traditional methods have not established an end-to-end, learnable collaborative mapping mechanism between the frequency and spatial domains. The injection of frequency components is still a shallow "rule-driven" operation, ignoring the deep interaction and adaptive weight allocation of features in the frequency and spatial domains. The aforementioned problems indicate that while wavelet-based or other fixed multi-resolution methods can improve spatial detail, they often struggle to maintain spectral fidelity and cross-scene stability. Therefore, this invention provides a frequency-spatial domain fusion panchromatic sharpening method, system, and storage medium based on DCT to address these technical issues. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a frequency-spatial domain fusion panchromatic sharpening method, system and storage medium based on DCT, which is used to solve the technical problems of difficulty in achieving lossless modal alignment between panchromatic images and multispectral images in image fusion, and difficulty in generating multispectral images that maintain spectral fidelity and have high spatial resolution.

[0005] To achieve the above objectives, a first aspect of the present invention provides a frequency-spatial domain fusion panchromatic sharpening method based on DCT, comprising: S1: Receive the target image; wherein the target image includes a high-resolution panchromatic image and a low-resolution multispectral image; S2: Extract dual-domain feature maps from the target image and stitch them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; S3: The dual-domain feature map after modality alignment is enhanced by channel attention mechanism, and the dual-domain feature map after feature enhancement is fused by multi-scale to obtain the first fused feature map; S4: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; perform feature enhancement processing on the frequency domain feature map corresponding to the first fused feature map and the first fused feature map through the channel attention mechanism, and perform multi-scale fusion on the frequency domain feature map after feature enhancement and the first fused feature map to obtain the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image.

[0006] Preferably, receiving the target image includes: High-resolution panchromatic images and low-resolution multispectral images are received via split input.

[0007] Preferably, the step of extracting dual-domain feature maps from the target image and stitching them together using DCT fusion to achieve modal alignment of the dual-domain feature maps includes: A1: Divide the high-resolution panchromatic image into non-overlapping 4×4 pixel blocks according to the resolution ratio of the low-resolution multispectral image; A2: Perform a 4-point two-dimensional discrete cosine transform on each 4×4 pixel block, and convert the pixel values ​​in the spatial domain into a frequency coefficient matrix containing one DC component and 15 AC components through matrix operations. A3: Use Pixel Unshuffle to rearrange the 16 frequency coefficient matrices calculated for each pixel block from the spatial dimension to the channel dimension; A4: Modal alignment is achieved by concatenating the frequency domain feature map of the transformed high-resolution panchromatic image with the spatial domain feature map of the low-resolution multispectral image along the channel dimension.

[0008] Preferably, the feature enhancement processing of the dual-domain feature map after modality alignment through a channel attention mechanism includes: Extract frequency domain feature maps and spatial domain feature maps, and perform global average pooling on the frequency domain feature maps and spatial domain feature maps to compress the two-dimensional features of each channel into a real number descriptor; Several real number descriptors are input into a multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated by the Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the spatial domain feature map.

[0009] Preferably, the step of performing multi-scale fusion on the dual-domain feature map after feature enhancement processing to obtain the first fused feature map includes: After feature enhancement processing, frequency domain feature maps and spatial domain feature maps are extracted from the dual-domain feature map. The spatial domain feature map of the low-resolution multispectral image is simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5 and 7×7, respectively. The multi-scale features include point details, local texture and global structural information. The multi-scale features are spliced ​​and fused along the channel dimension. The spliced ​​and fused spatial domain feature map is then further integrated with the frequency domain feature map that has undergone 3×3 convolution to obtain the first fused feature map.

[0010] Preferably, the feature enhancement processing of the frequency domain feature map corresponding to the first fused feature map and the first fused feature map through a channel attention mechanism includes: Extract the frequency domain feature map and the first fused feature map, and perform global average pooling on the frequency domain feature map and the first fused feature map to compress the two-dimensional features of each channel into a real number descriptor; Several real number descriptors are input into a multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated by the Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the first fused feature map.

[0011] Preferably, the step of performing multi-scale fusion of the frequency domain feature map after feature enhancement processing and the first fused feature map to obtain the second fused feature map includes: The frequency domain feature map after feature enhancement is extracted and combined with the first fused feature map. The first fused feature map is then simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5, and 7×7, respectively. The multi-scale features include point details, local textures, and global structural information. The multi-scale features are spliced ​​and fused along the channel dimension. The first fused feature map after splicing and fusion is further integrated with the frequency domain feature map after 3×3 convolution to obtain the second fused feature map.

[0012] Preferably, the step of residually concatenating the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image includes: The first fused feature map is upsampled to the same high-resolution size as the second fused feature map by an upsampling operation, such as bilinear interpolation. The second fused feature map is added pixel by pixel to the first fused feature map to obtain the third fused feature map. The third fused feature map is further fused through a convolutional layer to eliminate potential conflicts and enhance the consistency of spatial and spectral information. Finally, the high-resolution multispectral image is obtained by upsampling or deconvolution to restore it to the target resolution. The target resolution is set manually.

[0013] A second aspect of the present invention provides a frequency-spatial domain fusion panchromatic sharpening system based on DCT, comprising: Input module: Receives target images; wherein, the target images include high-resolution panchromatic images and low-resolution multispectral images; Dual-domain feature alignment module: Extracts dual-domain feature maps from the target image and stitches them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; Progressive refinement of modules: Step B1: Input the modally aligned dual-domain feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the first fused feature map, including: performing feature enhancement processing on the modally aligned dual-domain feature map through a channel attention mechanism, and performing multi-scale fusion on the feature-enhanced dual-domain feature map to obtain the first fused feature map; Step B2: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; input the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image; wherein, inputting the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and marking the fused feature map as the second fused feature map, includes: performing feature enhancement processing on the frequency domain feature map and the first fused feature map through a channel attention mechanism, and performing multi-scale fusion on the frequency domain feature map after feature enhancement processing and the first fused feature map to obtain the second fused feature map; The cross-domain feature fusion module is used to perform feature enhancement and multi-scale fusion on the two input feature maps to obtain a fused feature map.

[0014] A third aspect of the present invention provides a storage medium for storing a computer program, which, when executed, implements a frequency-spatial domain fusion pancolor sharpening method based on DCT as described in any of the first aspects.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention achieves modal alignment of the dual-domain feature maps by extracting dual-domain feature maps from the target image and stitching them together using DCT fusion. The modally aligned dual-domain feature maps are then enhanced using a channel attention mechanism, and multi-scale fusion is performed to obtain a first fused feature map. A DCT transformation is then performed on the first fused feature map to extract its corresponding frequency domain feature map. The frequency domain feature map corresponding to the first fused feature map and the first fused feature map are then enhanced using a channel attention mechanism, and multi-scale fusion is performed on the enhanced frequency domain feature map and the first fused feature map to obtain a second fused feature map. Residual concatenation of the second and first fused feature maps yields a high-resolution multispectral image. This invention solves the technical problems of achieving lossless modal alignment between panchromatic and multispectral images in image fusion, and the difficulty in generating multispectral images that maintain both spectral fidelity and high spatial resolution. This invention improves spatial detail while also ensuring spectral fidelity and cross-scene stability.

[0016] 2. The present invention proposes a frequency-spatial domain fusion panchromatic sharpening method based on DCT. Through a progressive thinning strategy and a two-stage cascaded thinning mechanism, it achieves efficient fusion of high-resolution panchromatic images and low-resolution multispectral images. While maintaining the spectral characteristics of multispectral images, this method can accurately recover high-frequency details and significantly enhance spatial structural information such as edges and textures. It provides an efficient and reliable panchromatic sharpening solution for remote sensing image processing, computer vision and other fields. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the operation steps of the present invention; Figure 2 This is a schematic diagram illustrating the operational steps for achieving modal alignment of dual-domain feature maps according to the present invention; Figure 3 This is a schematic diagram of the system modules of the present invention. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 The first aspect of this invention provides a frequency-spatial domain fusion panchromatic sharpening method based on DCT, comprising: S1: Receive the target image; wherein the target image includes a high-resolution panchromatic image and a low-resolution multispectral image; S2: Extract dual-domain feature maps from the target image and stitch them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; S3: The dual-domain feature map after modality alignment is enhanced by channel attention mechanism, and the dual-domain feature map after feature enhancement is fused by multi-scale to obtain the first fused feature map; S4: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; perform feature enhancement processing on the frequency domain feature map corresponding to the first fused feature map and the first fused feature map through the channel attention mechanism, and perform multi-scale fusion on the frequency domain feature map after feature enhancement and the first fused feature map to obtain the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image.

[0021] It is worth noting that the frequency-spatial domain fusion panchromatic sharpening method proposed in this invention achieves efficient fusion of high-resolution panchromatic images and low-resolution multispectral images through a progressive refinement strategy and a two-stage cascaded refinement mechanism, significantly improving the spatial detail quality and spectral fidelity of the fused image. This method first extracts dual-domain features from the input image, fully utilizing the advantages of discrete cosine transform in frequency domain analysis and combining it with spatial domain features to construct a multi-dimensional feature representation. DCT fusion is then used to achieve modal alignment of the dual-domain features, ensuring effective synergy between frequency and spatial information from the initial stage. In the first stage, a channel attention mechanism is used to enhance the dual-domain features, adaptively highlighting key frequency bands and spatial structure information. Then, multi-scale fusion technology is used to integrate feature details at different scales to form the first fused feature map. This stage mainly focuses on restoring the basic spatial structure and enhancing coarse-scale details, laying the foundation for subsequent refinement. The second stage further applies DCT transformation to the first fused feature map to extract deep frequency domain features. A second channel attention mechanism is then used to enhance high-frequency details and edge information. Multi-scale fusion technology is combined to supplement any fine-grained textures that might be missed in the spatial domain, generating a second fused feature map. Finally, the second and first fused feature maps are fused through residual connections. This fully leverages the complementary advantages of enhanced features learned by the network and the upsampled image, ensuring basic fidelity of spectral information while significantly improving the clarity and realism of spatial details. This coarse-to-fine progressive optimization design not only effectively solves the contradiction between spectral distortion and loss of spatial details in traditional panchromatic sharpening methods but also achieves layered mining and refined reconstruction of image features through a two-stage cascade mechanism, ensuring that the output of each stage is an effective improvement over the previous stage. This method, while maintaining the spectral characteristics of multispectral images, can accurately recover high-frequency details and significantly enhance spatial structural information such as edges and textures, providing an efficient and reliable panchromatic sharpening solution for remote sensing image processing, computer vision, and other fields.

[0022] It should be noted that this invention constructs an end-to-end deep learning network architecture called FreqDCT-Pan, whose internal processing flow is as follows: First, the network receives a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) as dual inputs. For the PAN image, spatial information is converted into frequency domain features through a 4×4 discrete cosine transform (DCT), and the size is adjusted using a pixel unshuffle operation; for the LRMS image, spatial domain features are extracted through convolutional layers. Subsequently, a dual-domain feature extraction and alignment module (DFE) concatenates these two features along the channel dimension to achieve modal alignment. The aligned features enter a cascaded cross-domain feature fusion module (CFF), which uses the high-frequency information of the panchromatic image to guide the fusion of multispectral features. After multi-layer fusion, the features finally enter a progressive refinement strategy (PRS) module, which gradually recovers spatial details through two-stage residual learning, ultimately outputting a high-resolution multispectral image (HRMS).

[0023] In establishing the end-to-end deep learning network architecture of FreqDCT-Pan, a combined optimization strategy of pixel-level L1 loss and spectral angle mapping SAM loss was also used for network architecture training. The specific steps are as follows: Calculate the pixel-level L1 loss between the predicted image generated by the network and the real reference image to ensure consistency of structure and texture; Simultaneously, the spectral angle mapping (SAM) loss between the predicted image and the real reference image on the spectral vector is calculated to measure the degree of spectral distortion; Two hyperparameters (lambda1 and lambda2) are set, and the L1 loss and SAM loss are weighted and summed to obtain the total loss. During training, the spatial sharpening and spectral preservation capabilities of the network are optimized simultaneously by minimizing the total loss. The formula for the total loss obtained by weighting and summing the L1 loss and SAM loss is: Total Loss = lambda1 × 3L1 Loss + lambda2 × SAM Loss; 3L1 Loss is three times the L1 loss, specifically 3 × L1 Loss, and SAM Loss is the spectral angle mapping loss.

[0024] It is worth noting that L1 loss ensures pixel-level consistency between the generated image and the real image, guaranteeing the fidelity of structure and texture, and exhibits good robustness to outliers, effectively avoiding over-smoothing. SAM loss is specifically designed for the spectral characteristics of multispectral images, measuring spectral fidelity by calculating the angular differences between spectral vectors, ensuring that the spectral characteristics of the fused image remain consistent with the real image. SAM loss is sensitive to the orientation of spectral vectors, effectively preserving the spectral feature differences of different land features. The two loss functions are combined in a weighted manner to achieve an optimal balance between spatial detail enhancement and spectral information preservation. The training process employs the Adam optimizer, sets an appropriate learning rate decay strategy, and uses data augmentation techniques, including random cropping, flipping, and rotation, to improve the model's generalization ability and robustness.

[0025] For example, taking panchromatic sharpening of WorldView-2 satellite imagery as an example: First, a single-channel high-resolution panchromatic image of 1024x1024 pixels and an 8-channel low-resolution multispectral image of 256x256 pixels are acquired. After preprocessing, the panchromatic image is transformed into a frequency feature map through a 4x4 block DCT transform and rearranged to a size of 256x256. The multispectral image is used to extract spatial features through 3x3, 5x5, and 7x7 convolutions. After channel concatenation, both images enter the dual-domain feature extraction module. Subsequently, the features enter the cross-domain fusion module, which uses a channel attention mechanism to adaptively enhance the weights of edge channels. Then, progressive refinement is performed: the first stage initially restores the building outlines, and the second stage performs another DCT transform on the preliminary results to extract high-frequency coefficients and re-inject them into the network, thereby enhancing fine textures such as roof tiles. Finally, the output residual image is added to the upsampled image to generate a 1024x1024 pixel, 8-channel high-fidelity, high-resolution multispectral image.

[0026] Receiving a target image in this application includes: High-resolution panchromatic images and low-resolution multispectral images are received via split input.

[0027] Please see Figure 2 In this application, dual-domain feature maps are extracted from the target image, and then stitched together using DCT fusion to achieve modal alignment of the dual-domain feature maps, including: A1: Divide the high-resolution panchromatic image into non-overlapping 4×4 pixel blocks according to the resolution ratio of the low-resolution multispectral image; A2: Perform a 4-point two-dimensional discrete cosine transform on each 4×4 pixel block, and convert the pixel values ​​in the spatial domain into a frequency coefficient matrix containing one DC component and 15 AC components through matrix operations. A3: Use Pixel Unshuffle to rearrange the 16 frequency coefficient matrices calculated for each pixel block from the spatial dimension to the channel dimension; A4: Modal alignment is achieved by concatenating the frequency domain feature map of the transformed high-resolution panchromatic image with the spatial domain feature map of the low-resolution multispectral image along the channel dimension.

[0028] It should be noted that the dual-domain feature extraction and alignment module abandons the traditional simple upsampling or downsampling alignment method, and instead uses DCT transform to achieve precise modal alignment. For panchromatic images, discrete cosine transform is used to extract rich frequency domain features, transforming the image from the spatial domain to the frequency domain, and fully utilizing the advantages of DCT transform in image compression and frequency analysis. DCT transform can decompose the image signal into different frequency components. The low-frequency components contain the main structural information of the image, while the high-frequency components contain edge and texture details. For multispectral images, a multi-scale convolutional architecture is used to capture spatial features of different receptive fields, and three different sizes of convolutional kernels are used to extract local details, medium-scale structure, and global contextual information, respectively. The extracted frequency domain features and spatial domain features are aligned and integrated through fusion, achieving an effective combination of information from the two domains and avoiding the information loss problem in traditional methods. In this study, three different sizes of convolutional kernels were used to extract local details, medium-scale structures, and global contextual information. The three kernel sizes used were 3×3, 5×5, and 7×7. The extraction method employed a parallel multi-scale convolutional architecture, where the input features passed through these three convolutional layers of different sizes simultaneously to capture local details (3×3), medium-scale structures (5×5), and global contextual information (7×7) of the image, respectively. Finally, the feature maps output by these three convolutional layers were concatenated along the channel dimension to obtain rich spatial features containing multi-scale receptive fields.

[0029] It should be noted that the resolution ratio of a high-resolution panchromatic image to a low-resolution multispectral image is typically four times.

[0030] It should be noted that Pixel Unshuffle is a pixel rearrangement.

[0031] It should be noted that by using Pixel Unshuffle to rearrange the 16 frequency coefficient matrices calculated for each pixel block from the spatial dimension to the channel dimension, this operation reduces the spatial resolution of the feature map to 1 / 4 of its original value, thus making it completely consistent with the low-resolution multispectral image in terms of spatial size.

[0032] It should be noted that by stitching the frequency domain feature map of the transformed high-resolution panchromatic image with the spatial domain feature map of the low-resolution multispectral image along the channel dimension, precise alignment in physical size is achieved, avoiding the loss of high-frequency information caused by traditional downsampling methods.

[0033] It should be noted that frequency domain features refer to the transform coefficients obtained by applying a 4-point two-dimensional discrete cosine transform (DCT) to a high-resolution panchromatic image (PAN). These coefficients are separated into low-frequency components representing the overall structure and high-frequency components representing edge texture. Spatial domain features refer to the pixel-level feature maps directly extracted by applying a multi-scale convolutional network to a low-resolution multispectral image (LRMS). Alignment is achieved through the energy compression property of the DCT transform. That is, after performing a block-wise DCT transform on the high-resolution panchromatic image, the size of its frequency coefficient matrix is ​​naturally reduced in the spatial dimension, allowing it to be matched with the low-resolution multispectral feature map without destructive downsampling. Subsequently, the frequency domain feature map of the high-resolution panchromatic image and the spatial domain feature map of the multispectral image are directly concatenated in the channel dimension, thereby achieving lossless modal alignment.

[0034] In this application, the dual-domain feature map after modality alignment is enhanced using a channel attention mechanism, including: Extract frequency domain feature maps and spatial domain feature maps, and perform global average pooling on the frequency domain feature maps and spatial domain feature maps to compress the two-dimensional features of each channel into a real number descriptor. Several real-valued descriptors are input into the multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated using the Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the spatial domain feature map.

[0035] It should be noted that the dual-domain feature maps after modality alignment are enhanced by a channel attention mechanism. This mechanism can automatically assign higher weights to channels with rich edge textures based on the image content, while suppressing channels containing redundant information or noise. This enhancement mechanism can automatically identify the feature channels most useful for the current task, thereby improving the network's expressive and discriminative capabilities.

[0036] In this application, a first fused feature map is obtained by multi-scale fusion of the dual-domain feature map after feature enhancement processing, including: After feature enhancement processing, frequency domain feature maps and spatial domain feature maps are extracted from the dual-domain feature map. The spatial domain feature map of the low-resolution multispectral image is simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5 and 7×7, respectively. The multi-scale features include point details, local texture and global structural information. Multi-scale features are spliced ​​and fused along the channel dimension. The spliced ​​and fused spatial domain feature map is then further integrated with the frequency domain feature map that has undergone 3×3 convolution to obtain the first fused feature map.

[0037] It's important to note that, to fully capture contextual information at different scales, a parallel multi-scale convolutional structure was designed. Low-resolution features are not processed individually but are simultaneously input into four parallel convolutional branches with kernel sizes of 1×1, 3×3, 5×5, and 7×7. These convolutional layers with different receptive fields can extract point details, local texture, and global structural information, respectively. The extracted multi-scale features are then concatenated along the channel dimension, and the fused features are further interacted with and integrated with the frequency domain feature map processed by the 3×3 convolution. This design ensures that the network can simultaneously capture both the macroscopic structure and microscopic details of the image.

[0038] In this application, the frequency domain feature map corresponding to the first fused feature map and the first fused feature map are subjected to feature enhancement processing through a channel attention mechanism, including: Extract the frequency domain feature map and the first fused feature map, and perform global average pooling on the frequency domain feature map and the first fused feature map to compress the two-dimensional features of each channel into a real number descriptor; Several real-valued descriptors are input into the multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated using the Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the first fused feature map.

[0039] It is worth noting that this step constructs a dynamic and adaptive feature refinement loop. Through global average pooling, each two-dimensional feature channel is compressed into a real-valued descriptor with a global receptive field. This operation efficiently aggregates information across spatial dimensions, forming a macroscopic understanding of the importance of each channel. Subsequently, these descriptors undergo nonlinear transformation via a multilayer perceptron to capture the complex interdependencies between channels, identifying the synergistic effects of different feature combinations and avoiding the simplification of complex relationships by linear models. The resulting channel weight coefficients selectively enhance the key channels that contribute most to spatial detail recovery and spectral information preservation, while effectively suppressing or weakening secondary channels containing noise or redundant information, based on the dynamic changes in the input content. This adaptive feature recalibration process allows the network to focus on the most discriminative information, greatly improving the feature expression efficiency and robustness. It ensures that the fused feature map retains rich spatial texture while maintaining the spectral integrity of the multispectral image, providing accurate feature assurance for generating high-quality panchromatic sharpened images.

[0040] In this application, a second fused feature map is obtained by multi-scale fusion of the frequency domain feature map after feature enhancement and the first fused feature map, including: The frequency domain feature map after feature enhancement is extracted and combined with the first fused feature map. The first fused feature map is then simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5, and 7×7, respectively. The multi-scale features include point details, local textures, and global structural information. Multi-scale features are spliced ​​and fused along the channel dimension. The first fused feature map is then further integrated with the frequency domain feature map that has undergone 3×3 convolution to obtain the second fused feature map.

[0041] In this application, a high-resolution multispectral image is obtained by residually connecting the second fused feature map and the first fused feature map, including: The first fused feature map is upsampled to the same high-resolution size as the second fused feature map by an upsampling operation, such as bilinear interpolation. The second fused feature map is added pixel by pixel to the first fused feature map to obtain the third fused feature map. The third fused feature map is further fused through a convolutional layer to eliminate potential conflicts and enhance the consistency of spatial and spectral information. Finally, the high-resolution multispectral image is obtained by upsampling or deconvolution to restore it to the target resolution. The target resolution is set manually.

[0042] Please see Figure 3 A second aspect of the present invention provides a frequency-spatial domain fusion full-color sharpening system based on DCT, comprising: Input module: Receives target images; wherein, the target images include high-resolution panchromatic images and low-resolution multispectral images; Dual-domain feature alignment module: Extracts dual-domain feature maps from the target image and stitches them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; Progressive refinement of modules: Step B1: Input the modally aligned dual-domain feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the first fused feature map, including: performing feature enhancement processing on the modally aligned dual-domain feature map through a channel attention mechanism, and performing multi-scale fusion on the feature-enhanced dual-domain feature map to obtain the first fused feature map; Step B2: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; input the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image; wherein, inputting the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and marking the fused feature map as the second fused feature map, includes: performing feature enhancement processing on the frequency domain feature map and the first fused feature map through a channel attention mechanism, and performing multi-scale fusion on the frequency domain feature map after feature enhancement processing and the first fused feature map to obtain the second fused feature map; The cross-domain feature fusion module is used to perform feature enhancement and multi-scale fusion on the two input feature maps to obtain a fused feature map.

[0043] A third aspect of the present invention provides a storage medium for storing a computer program, which, when executed, implements a frequency-spatial domain fusion pancolor sharpening method based on DCT as described in any of the first aspects.

[0044] Working principle of the invention: This invention proposes a frequency-spatial domain fusion panchromatic sharpening method based on Discrete Cosine Transform (DCT), employing a deep neural network architecture to achieve high-quality image fusion. The core idea is to utilize the energy compression and frequency separation characteristics of DCT to achieve lossless modal alignment between panchromatic and multispectral images. Furthermore, through dual-domain feature extraction and cross-domain feature fusion techniques, a multispectral image that maintains both spectral fidelity and high spatial resolution is generated.

[0045] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A frequency-spatial domain fusion panchromatic sharpening method based on DCT, characterized in that, include: S1: Receive the target image; wherein the target image includes a high-resolution panchromatic image and a low-resolution multispectral image; S2: Extract dual-domain feature maps from the target image and stitch them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; S3: The dual-domain feature map after modality alignment is enhanced by channel attention mechanism, and the dual-domain feature map after feature enhancement is fused by multi-scale to obtain the first fused feature map; S4: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; perform feature enhancement processing on the frequency domain feature map corresponding to the first fused feature map and the first fused feature map through the channel attention mechanism, and perform multi-scale fusion on the frequency domain feature map after feature enhancement and the first fused feature map to obtain the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image.

2. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The step of extracting dual-domain feature maps from the target image and stitching them together using DCT fusion to achieve modal alignment of the dual-domain feature maps includes: A1: Divide the high-resolution panchromatic image into non-overlapping 4×4 pixel blocks according to the resolution ratio of the low-resolution multispectral image; A2: Perform a 4-point two-dimensional discrete cosine transform on each 4×4 pixel block, and convert the pixel values ​​in the spatial domain into a frequency coefficient matrix containing one DC component and 15 AC components through matrix operations. A3: Use Pixel Unshuffle to rearrange the 16 frequency coefficient matrices calculated for each pixel block from the spatial dimension to the channel dimension; A4: The frequency domain feature map of the transformed high-resolution panchromatic image is stitched together with the spatial domain feature map of the low-resolution multispectral image along the channel dimension.

3. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The feature enhancement process performed on the dual-domain feature map after modality alignment using a channel attention mechanism includes: Extract frequency domain feature maps and spatial domain feature maps, and perform global average pooling on the frequency domain feature maps and spatial domain feature maps to compress the two-dimensional features of each channel into a real number descriptor; Several real-valued descriptors are input into a multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated using a Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the spatial domain feature map.

4. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The process of multi-scale fusion of the feature-enhanced dual-domain feature maps to obtain the first fused feature map includes: After feature enhancement processing, frequency domain feature maps and spatial domain feature maps are extracted from the dual-domain feature map. The spatial domain feature map of the low-resolution multispectral image is simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5 and 7×7, respectively. The multi-scale features include point details, local texture and global structural information. The multi-scale features are spliced ​​and fused along the channel dimension. The spliced ​​and fused spatial domain feature map is then further integrated with the frequency domain feature map that has undergone 3×3 convolution to obtain the first fused feature map.

5. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The feature enhancement process performed on the frequency domain feature map corresponding to the first fused feature map and the first fused feature map through a channel attention mechanism includes: Extract the frequency domain feature map and the first fused feature map, and perform global average pooling on the frequency domain feature map and the first fused feature map to compress the two-dimensional features of each channel into a real number descriptor; Several real-valued descriptors are input into a multilayer perceptron to capture the nonlinear dependencies between channels. Based on the nonlinear dependencies, channel weight coefficients ranging from 0 to 1 are generated using a Sigmoid activation function. The weighting coefficients are multiplied element-wise with the corresponding channels of the frequency domain feature map and the first fused feature map.

6. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The process of fusing the frequency domain feature map after feature enhancement with the first fused feature map at multiple scales to obtain the second fused feature map includes: The frequency domain feature map after feature enhancement is extracted and combined with the first fused feature map. The first fused feature map is then simultaneously input into four parallel convolutional branches to extract multi-scale features. The convolutional kernel sizes of the convolutional branches are 1×1, 3×3, 5×5, and 7×7, respectively. The multi-scale features include point details, local textures, and global structural information. The multi-scale features are spliced ​​and fused along the channel dimension. The first fused feature map after splicing and fusion is further integrated with the frequency domain feature map after 3×3 convolution to obtain the second fused feature map.

7. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The step of residually concatenating the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image includes: The first fused feature map is upsampled to the same high-resolution size as the second fused feature map; The second fused feature map is added pixel by pixel to the first fused feature map to obtain the third fused feature map. The third fused feature map is further fused through a convolutional layer, and finally the high-resolution multispectral image is obtained by upsampling or deconvolution to restore it to the target resolution. The target resolution is determined manually.

8. The frequency-spatial domain fusion panchromatic sharpening method based on DCT according to claim 1, characterized in that, The received target image includes: High-resolution panchromatic images and low-resolution multispectral images are received via split input.

9. A frequency-spatial domain fusion pancolor sharpening system based on DCT, used to run the frequency-spatial domain fusion pancolor sharpening method based on DCT as described in any one of claims 1 to 8, characterized in that, include: Input module: Receives target images; wherein, the target images include high-resolution panchromatic images and low-resolution multispectral images; Dual-domain feature alignment module: Extracts dual-domain feature maps from the target image and stitches them together using DCT fusion to achieve modal alignment of the dual-domain feature maps; wherein, the dual-domain feature maps include frequency domain feature maps and spatial domain feature maps; Progressive refinement of modules: Step B1: Input the modally aligned dual-domain feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the first fused feature map, including: performing feature enhancement processing on the modally aligned dual-domain feature map through a channel attention mechanism, and performing multi-scale fusion on the feature-enhanced dual-domain feature map to obtain the first fused feature map; Step B2: Perform DCT transformation on the first fused feature map to extract the frequency domain feature map corresponding to the first fused feature map; input the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and mark the fused feature map as the second fused feature map; perform residual connection on the second fused feature map and the first fused feature map to obtain a high-resolution multispectral image; wherein, inputting the frequency domain feature map corresponding to the first fused feature map and the first fused feature map into the cross-domain feature fusion module to obtain a fused feature map, and marking the fused feature map as the second fused feature map, includes: performing feature enhancement processing on the frequency domain feature map and the first fused feature map through a channel attention mechanism, and performing multi-scale fusion on the frequency domain feature map after feature enhancement processing and the first fused feature map to obtain the second fused feature map; The cross-domain feature fusion module is used to perform feature enhancement and multi-scale fusion on the two input feature maps to obtain a fused feature map.

10. A storage medium, characterized in that, Used to store a computer program, which, when executed, implements the frequency-spatial domain fusion pancolor sharpening method based on any one of claims 1 to 8.

Images