A high-fidelity panchromatic sharpening method and system for large-format remote sensing images

Abstract

This application relates to the fields of computer vision and multimodal image processing, and specifically to a high-fidelity panchromatic sharpening method and system for large-format remote sensing images. The method first upsamples the LRMS (Local Level Metric Image) and then inputs the upsampled multispectral image and PAN (Polyspectral Image) into the multispectral coding branch and panchromatic coding branch respectively for multi-stage hierarchical feature extraction. Subsequently, a staged gating bidirectional exchange mechanism is introduced to predict two spatially gated images. High-frequency detail information from the PAN branch is adaptively injected into the MS branch, while low-frequency spectral context from the MS branch is fed back to the PAN branch, simultaneously generating fused skip-connection features. Then, unified forward computation is performed on the large-format input regions or markers. Finally, high-resolution multispectral HRMS output is obtained through coarse-to-fine decoding and residual reconstruction. This application balances spatial detail enhancement, spectral fidelity, and large-format deployment efficiency, and is suitable for applications such as remote sensing mapping, fine-grained ground feature identification, urban monitoring, agricultural observation, and disaster assessment.

Description

Technical Field

[0001] This application relates to the fields of computer vision and multimodal image processing, and in particular to a high-fidelity panchromatic sharpening method and system for large-format remote sensing images. Background Technology

[0002] Due to limitations in sensor physical conditions, remote sensing satellites typically struggle to simultaneously acquire observations with both high spatial and high spectral resolution. Panchromatic images offer high spatial resolution, reflecting rich texture, edge, and structural details; multispectral images provide good spectral fidelity, reflecting the spectral distribution characteristics of ground features, but their spatial resolution is relatively low. Panchromatic sharpening technology aims to combine the advantages of both types of images to generate high-resolution multispectral images that possess both high spatial resolution and high spectral fidelity. This technology has been widely applied in remote sensing scenarios such as land use classification, target recognition, change detection, detailed mapping, disaster monitoring, and resource surveys, and is of great significance for improving the interpretation capabilities and application value of remote sensing imagery.

[0003] While existing pancolor sharpening methods have achieved some success on standard-sized images, most methods are still limited to training and validation on small image patches. In practical deployment, it is often necessary to perform external patch inference on large-format remote sensing images such as 4K and 8K, and then stitch the output results of each patch into a complete image. This processing method is prone to insufficient interaction between patch boundaries, resulting in problems such as local discontinuities, stitching artifacts, and spatial structure fragmentation. At the same time, directly inferring from the entire large-format image leads to excessive memory consumption, making it difficult to deploy on mainstream GPU platforms. In addition, large-format remote sensing scenes usually cover a wider area, have more complex land cover types, and more obvious scale differences between regions. Traditional methods based on small patch training and local receptive field modeling often fail to maintain sufficient global structural consistency and cross-regional contextual continuity under large-scene conditions, thus affecting the overall quality of pancolor sharpening results.

[0004] Furthermore, existing methods for cross-modal information interaction often simply stitch together the multispectral and panchromatic branches or fuse them all at once during the decoding stage. This lacks differentiated control over the division of labor between "panchromatic detail injection" and "multispectral spectral context feedback," easily leading to over-injection of details resulting in spectral distortion, or good spectral preservation but insufficient spatial detail. While some methods attempt to enhance fusion performance by introducing multi-scale modeling, attention mechanisms, or complex transformation structures, this also introduces higher computational overhead and memory burden, further increasing the difficulty of deployment under large-format input conditions. Therefore, there is an urgent need to propose a panchromatic sharpening method that can process large-format remote sensing images under the constraints of mainstream hardware resources while simultaneously achieving both spatial detail enhancement and spectral fidelity. Summary of the Invention

[0005] This application provides a high-fidelity panchromatic sharpening method and system for large-format remote sensing images to solve the problems of high memory usage, easy generation of boundary artifacts by external block stitching, insufficient cross-modal information interaction, and difficulty in balancing spatial detail and spectral fidelity in existing methods for large-format scenes.

[0006] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a high-fidelity panchromatic sharpening method for large-format remote sensing images, comprising the following steps: Step S1: Acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN; Step S2: Input the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral features and panchromatic features at the corresponding scales of each stage; Step S3: Perform staged gated bidirectional exchange between each coding stage, perform gated cross-modal interaction on the multispectral features and panchromatic features, and generate a decoder while completing the cross-modal interaction. The fusion jump-connection features used; Step S4: Within the network, an asynchronous storage-aware sequential processing method oriented towards large-format input is adopted to perform unified forward computation on the divided partitioned regions or labels, so that different regions can complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction; Step S5: The fusion jump-connection features obtained after staged gated bidirectional exchange are input into the decoder, and coarse-to-fine decoding and reconstruction are performed in the order from deep to shallow. At each decoding stage, the fusion jump-connection features of the corresponding scale are combined to restore spatial details and spectral information, and finally output high-resolution multispectral residual R. The residual reconstruction is performed between R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output.

[0007] In some exemplary embodiments, step S1 specifically includes: step S101, acquiring a high-resolution panchromatic image PAN and a corresponding low-resolution multispectral image LRMS for the same region; step S102, upsampling the LRMS using a "23-tap polynomial" interpolation method to ensure that the upsampled multispectral image maintains the same spatial size as the PAN; step S103, mapping the LRMS to the spatial resolution corresponding to the PAN without changing the original multispectral band information, thereby providing a unified input for subsequent cross-modal feature extraction and panchromatic sharpening reconstruction.

[0008] In some exemplary embodiments, step S2 specifically includes: step S201, inputting the upsampled multispectral image into the multispectral coding branch and the PAN into the panchromatic coding branch; step S202, using a feature extraction backbone network with the same hierarchical structure but different input channel numbers to perform multi-stage hierarchical feature extraction on the upsampled multispectral image and the PAN respectively; step S203, using the multispectral coding branch to extract contextual information with spectral consistency and multi-scale semantic representation, and using the panchromatic coding branch to extract spatial texture detail information and structural edge information; step S204, the multispectral coding branch and the panchromatic coding branch perform feature extraction in a shallow to deep order, and each stage includes a feature transformation module, so that the multispectral features and panchromatic features gradually expand the receptive field and improve the semantic abstraction ability as the network depth increases; step S205, after each stage, output the corresponding scale of bimodal features, and the multispectral features and panchromatic features updated by gated bidirectional exchange in the current stage continue to be input into the next stage for subsequent stage-based gated bidirectional exchange and decoding reconstruction, and so that cross-modal interaction continues to affect the deeper representation learning process.

[0009] In some exemplary embodiments, in the s-th encoding stage, the multispectral features and panchromatic features are respectively represented as:

[0010] in, This represents the upsampled multispectral image. This represents the feature mapping of the multispectral coding branch at the s-th stage. This represents the feature mapping of the full-color coding branch at the s-th stage. This represents the multispectral features output at the s-th stage. This represents the panchromatic feature output at the s-th stage.

[0011] In some exemplary embodiments, the staged gated bidirectional exchange in step S3 includes: step S301, transferring multispectral features of the same stage. and panchromatic features Step S302: Concatenate the concatenated features along the channel dimension to form a joint representation feature; based on the concatenated features, predict two single-channel spatial gating maps using 1×1 convolution and the Sigmoid function respectively. and Among them, the spatial gating map is used to characterize the intensity of cross-modal information exchange at the current spatial location; step S303: by performing local average smoothing and subtraction on the panchromatic features, the detail enhancement component reflecting the spatial edge and texture changes is extracted. And according to the spatial gating diagram The detail enhancement component is adaptively injected into the multispectral features to obtain the updated multispectral features; step S304: by performing a local averaging operation on the multispectral features, a smooth contextual component reflecting spectral consistency and regional contextual relationships is extracted. And according to the spatial gating diagram The smoothed contextual component is fed back into the panchromatic feature to obtain the updated panchromatic feature; in step S305, the updated multispectral feature and the updated panchromatic feature are concatenated along the channel dimension and compressed to the preset channel dimension through a 1×1 convolution to obtain the fused skip-connected feature. Step S306: The updated multispectral features and updated panchromatic features are then passed to subsequent coding stages to achieve the step-by-step propagation of cross-modal information in the multi-stage hierarchical network.

[0012] In some exemplary embodiments, step S4 specifically includes: step S401, sequentially processing the corresponding partition regions or markers of the large-format input within the encoder, and completing the corresponding feature extraction and update; step S402, sequentially aggregating and progressively restoring the partition regions or markers within the decoder; step S403, completing feature update and aggregation between different regions within the forward path of the unified network, rather than independently completing block inference outside the network and then performing image stitching; step S404, reducing boundary stitching artifacts and maintaining the continuity of the large scene context through asynchronous storage-aware sequential processing.

[0013] In some exemplary embodiments, step S5 specifically includes: step S501, using the deepest fused skip-connected feature as the initial input to the decoder; step S502, concatenating the current decoded feature with the corresponding scale fused skip-connected feature at each decoding stage to obtain the concatenated feature; step S503, performing upsampling and feature recovery on the concatenated feature to reconstruct the spatial resolution step by step; step S504, finally outputting the high-resolution multispectral residual R through the convolutional layer; step S505, performing residual reconstruction with the upsampled multispectral image to obtain the full-color sharpening output.

[0014] Secondly, this application also provides a high-fidelity panchromatic sharpening system for large-format remote sensing images. This system implements the high-fidelity panchromatic sharpening method for large-format remote sensing images described in the above embodiments. The system includes: an upsampling module, a feature extraction module, a gated cross-modal interaction module, and a reconstruction module connected sequentially. The upsampling module acquires a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, upsamples the LRMS, and obtains an upsampled multispectral image with the same spatial size as the PAN. The feature extraction module inputs the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch, respectively, to perform multi-stage hierarchical feature extraction, obtaining multispectral and panchromatic features at corresponding scales for each stage. The gated cross-modal interaction module performs staged gated bidirectional exchange between coding stages, reconstructing the multispectral image. The system performs gated cross-modal interaction on spectral and panchromatic features, and generates fused skip-connected features for the decoder while completing the cross-modal interaction. The reconstruction module inputs the fused skip-connected features obtained after staged gated bidirectional exchange into the decoder, and performs coarse-to-fine decoding and reconstruction in order from deep to shallow. At each decoding stage, the fused skip-connected features of the corresponding scale are combined to restore spatial details and spectral information, and finally outputs high-resolution multispectral residual R. The residual reconstruction is performed on R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output. The system adopts an asynchronous storage-aware sequential processing method for large-format input within the network, and performs unified forward computation on the divided partition regions or labels, so that different regions complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction.

[0015] In addition, this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the high-fidelity panchromatic sharpening method for large-format remote sensing images described in the above embodiments.

[0016] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the high-fidelity panchromatic sharpening method for large-format remote sensing images described in the above embodiments.

[0017] The technical solution provided in this application has at least the following advantages: This application provides a high-fidelity panchromatic sharpening method and system for large-format remote sensing images. The method includes the following steps: Step S1: Acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN; Step S2: Input the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral features and panchromatic features at the corresponding scales of each stage; Step S3: Perform stage-gated bidirectional exchange between each coding stage, perform gated cross-modal interaction on the multispectral features and panchromatic features, and generate a fusion for the decoder to use while completing the cross-modal interaction. Step S4: Within the network, an asynchronous storage-aware sequential processing method oriented towards large-format input is adopted to perform unified forward computation on the divided partitioned regions or labels, so that different regions can complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction; Step S5: The fused jump-connected features obtained after staged gated bidirectional exchange are input into the decoder, and coarse-to-fine decoding and reconstruction are performed in the order from deep to shallow. At each decoding stage, the fused jump-connected features of the corresponding scale are combined to restore spatial details and spectral information, and finally output high-resolution multispectral residual R. R is then used to perform residual reconstruction with the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output.

[0018] The high-fidelity panchromatic sharpening method for large-format remote sensing images provided in this application can achieve high-fidelity panchromatic sharpening reconstruction in large-format remote sensing image scenarios. The method extracts features from both high-resolution panchromatic and low-resolution multispectral images using a bi-branch hierarchical approach, modeling spatial texture details in the panchromatic image and spectral context information in the multispectral image, respectively. A staged gating bidirectional exchange mechanism is introduced at each encoding stage to achieve controlled injection of panchromatic details into the multispectral branch and feedback enhancement of the multispectral spectral context into the panchromatic branch, thereby improving spatial detail recovery and spectral consistency preservation. For large-format inputs such as 4K and 8K, the method further employs an asynchronous storage-aware sequential processing approach within the network, reducing peak memory usage while maintaining cross-regional hierarchical feature interaction, effectively reducing boundary stitching artifacts and context fragmentation problems caused by traditional external block inference. Through multi-scale fusion skip-connection feature-guided stepwise decoding and residual reconstruction, the spatial clarity, structural continuity, and spectral fidelity of the reconstruction results can be effectively improved. This method is applicable to various scenarios such as high-resolution remote sensing image interpretation, fine mapping, target recognition, change detection, disaster monitoring, and resource surveys, and has strong engineering practicality and promotional value. Attached Figure Description One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments, and unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 This is a flowchart illustrating a high-fidelity panchromatic sharpening method for large-format remote sensing images provided in an embodiment of this application.

[0020] Figure 2 This is a schematic diagram illustrating the specific process and asynchronous processing strategy of a high-fidelity panchromatic sharpening method for large-format remote sensing images provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of a staged gated bidirectional switching module structure provided in an embodiment of this application.

[0022] Figure 4a and Figure 4b This is a schematic diagram illustrating the qualitative comparison results of different methods under different dataset downresolution evaluations, as provided in an embodiment of this application.

[0023] Figure 5a and Figure 5b This is a schematic diagram illustrating the qualitative comparison results of different methods under full-resolution evaluation of different datasets, provided as an embodiment of this application.

[0024] Figure 6 This is a schematic diagram illustrating the qualitative comparison results of different methods in different 8K large-format scenarios with different datasets, provided in an embodiment of this application.

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] As can be seen from the background technology, existing methods have problems such as high memory usage, easy generation of boundary artifacts by external block splicing, insufficient cross-modal information interaction, and difficulty in balancing spatial detail and spectral fidelity in large-format scenarios.

[0027] To address the aforementioned technical problems, this application provides a high-fidelity panchromatic sharpening method and system for large-format remote sensing images. The method includes the following steps: Step S1: Acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region; upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN; Step S2: Input the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral and panchromatic features at corresponding scales for each stage; Step S3: Perform stage-gated bidirectional exchange between coding stages, perform gated cross-modal interaction on the multispectral and panchromatic features, and generate a data source for the decoder while completing the cross-modal interaction. The method utilizes fused skip-connected features. Step S4 involves employing an asynchronous storage-sensing sequential processing approach oriented towards large-format input within the network. This performs unified forward computation on the divided regions or markers, enabling different regions to complete feature extraction, cross-modal interaction, feature updating, and hierarchical aggregation within the same network computation path. This reduces peak memory usage while maintaining cross-regional hierarchical feature interaction. Step S5 involves inputting the fused skip-connected features obtained after staged gating and bidirectional exchange into the decoder. Coarse-to-fine decoding and reconstruction are performed in a deep-to-shallow order. At each decoding stage, the fused skip-connected features of the corresponding scale are combined to recover spatial details and spectral information. The final output is a high-resolution multispectral residual R. R is then reconstructed with the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output. This method enables high-fidelity panchromatic sharpening reconstruction in large-format remote sensing image scenarios.

[0028] The embodiments of this application will now be described in detail with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate a better understanding of the application. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0029] See Figure 1 This application provides a high-fidelity panchromatic sharpening method for large-format remote sensing images, including the following steps: Step S1: Acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN.

[0030] Step S2: Input the upsampled multispectral image and PAN into the multispectral coding branch and panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain the multispectral features and panchromatic features at the corresponding scales of each stage.

[0031] Step S3: Perform staged gated bidirectional exchange between each encoding stage, perform gated cross-modal interaction on multispectral and panchromatic features, and generate fused skip-connect features for the decoder to use while completing the cross-modal interaction.

[0032] Step S4: Within the network, an asynchronous storage-aware sequential processing method oriented towards large-format input is adopted to perform unified forward computation on the partitioned regions or labels, so that different regions can complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction.

[0033] Step S5: Input the fused skip-connection features obtained after staged gating bidirectional exchange into the decoder, and perform coarse-to-fine decoding and reconstruction in the order from deep to shallow. In each decoding stage, combine the fused skip-connection features of the corresponding scale to recover spatial details and spectral information, and finally output high-resolution multispectral residual R. Then, perform residual reconstruction with R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output.

[0034] Figure 2 The method flow and asynchronous processing strategy of this application are illustrated in the diagram. Specifically, step S1 upsamples the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN, thereby providing spatially scale-consistent input data for subsequent multimodal joint modeling while maintaining the original spectral distribution relationship of the multispectral images.

[0035] In some embodiments, step S1 specifically includes: Step S101: Acquire the high-resolution panchromatic image PAN and the corresponding low-resolution multispectral image LRMS for the same region. The two types of images correspond to each other in the observation area and are used as the raw input data for subsequent panchromatic sharpening processing.

[0036] Step S102: Upsample the LRMS using the “23-tap polynomial” interpolation method to ensure that the upsampled multispectral image is consistent with the PAN in spatial size, thereby meeting the requirements of input scale consistency for subsequent bi-branch feature extraction and cross-modal interaction.

[0037] Step S103: Without changing the original multispectral band information, map LRMS to the spatial resolution corresponding to PAN, thereby providing a unified input for subsequent cross-modal feature extraction and panchromatic sharpening reconstruction.

[0038] Specifically, step S103 mainly involves mapping the low-resolution multispectral image to the spatial resolution corresponding to the panchromatic image without changing the original multispectral band information, radiation distribution characteristics, and correspondence between bands. This provides a unified input for subsequent bi-branch feature extraction, detail injection, and panchromatic sharpening reconstruction, and minimizes unnecessary disturbances to the spectral information caused by the preprocessing stage.

[0039] The method provided in this application mainly involves multi-stage feature encoding in step S2 under the dual-branch hierarchical feature extraction framework to obtain multispectral and panchromatic features at the corresponding scales of each stage. This allows the multispectral branch to focus on maintaining spectral consistency and regional contextual semantic information between bands, while the panchromatic branch can focus on extracting spatial texture, edge, and structural detail information.

[0040] In some embodiments, step S2 specifically includes: Step S201: Input the upsampled multispectral image into the multispectral coding branch and input the PAN into the panchromatic coding branch, so that the two modal data are extracted along independent paths to avoid prematurely compressing the two into a single shared representation in the shallow stage.

[0041] Step S202: Using a feature extraction backbone network with the same hierarchical structure but different number of input channels, multi-stage hierarchical feature extraction is performed on the upsampled multispectral image and PAN respectively to obtain bimodal feature representations at different scales, and to provide a hierarchical feature foundation for subsequent multi-scale interaction and reconstruction.

[0042] Step S203: Use the multispectral coding branch to extract contextual information with spectral consistency and multi-scale semantic representation, and use the panchromatic coding branch to extract spatial texture detail information and structural edge information, so that the two modalities can complete the initial modeling in their respective more suitable representation spaces and highlight their respective advantages in the panchromatic sharpening task.

[0043] In step S204, the multispectral coding branch and the panchromatic coding branch extract features in order from shallow to deep. Each stage includes a feature transformation module, which allows the multispectral and panchromatic features to gradually expand their receptive field and improve their semantic abstraction ability as the network depth increases, thereby taking into account both local detail response and large-scale scene structure representation.

[0044] Step S205: After each stage is completed, output the bimodal features at the corresponding scale. The multispectral features and panchromatic features updated by the gated bidirectional exchange in the current stage are then input into the next stage for subsequent staged gated bidirectional exchange and decoding reconstruction. This allows cross-modal interaction to continuously influence the deeper representation learning process, thereby improving the overall performance of panchromatic sharpening results in terms of spatial detail and spectral consistency.

[0045] In some embodiments, during the s-th encoding stage, the multispectral features and panchromatic features are respectively represented as:

[0046] in, This represents the upsampled multispectral image. This represents the feature mapping of the multispectral coding branch at the s-th stage. This represents the feature mapping of the full-color coding branch at the s-th stage. This represents the multispectral features output at the s-th stage. This represents the panchromatic feature output at the s-th stage.

[0047] Figure 3 This is a schematic diagram of the staged gating bidirectional exchange module structure in step S3 of this application. Specifically, step S3 mainly uses a spatial gating mechanism to perform cross-modal interaction between multispectral and panchromatic features, enabling controlled injection of panchromatic detail information into the multispectral branch, while simultaneously providing feedback enhancement of multispectral spectral context information into the panchromatic branch. This achieves an adaptive balance between spatial detail enhancement and spectral fidelity preservation at different spatial locations, and generates fused skip-connection features for the decoder to use while completing the interaction.

[0048] In some embodiments, the staged gating bidirectional switching in step S3 includes: Step S301: Combine the multispectral features of the same stage and panchromatic features By splicing the data along the channel dimension, a joint representation feature is formed to fully integrate the complementary information of the two modalities at the current scale and to provide a joint discrimination basis for subsequent spatial gating weight prediction.

[0049] Step S302: Based on the concatenated features, predict two single-channel spatial gating maps using 1×1 convolution and the Sigmoid function respectively. and One spatial gating map controls the injection intensity of panchromatic detail information into multispectral features, while the other spatial gating map controls the feedback intensity of multispectral smoothing context information into panchromatic features, thus enabling the bidirectional information flow to have adaptive adjustment capabilities at different spatial locations.

[0050] Step S303: Extract detail enhancement components that reflect spatial edge and texture changes by performing local average smoothing and subtraction on the panchromatic features. And according to the spatial gating diagram By adaptively injecting detail enhancement components into multispectral features, updated multispectral features are obtained, enabling multispectral branches to achieve stronger spatial detail recovery capabilities while maintaining the original spectral representation.

[0051] Step S304: By performing a local averaging operation on the multispectral features, extract smooth contextual components that reflect spectral consistency and regional contextual relationships. And according to the spatial gating diagram By feeding back the smoothing context component into the panchromatic feature, we obtain the updated panchromatic feature, which allows the panchromatic branch to obtain more stable spectral context constraints while retaining the advantages of texture edges.

[0052] Step S305: Concatenate the updated multispectral features and the updated panchromatic features along the channel dimension, and compress them to the preset channel dimension through a 1×1 convolution to obtain the fused skip-connected features. This is so that the decoder can call it during the multi-scale reconstruction process.

[0053] Step S306: The updated multispectral features and updated panchromatic features are then passed to subsequent encoding stages to achieve the step-by-step propagation of cross-modal information in the multi-stage hierarchical network. This allows the interaction results at the current scale to continuously influence the formation process of deeper features, thereby enhancing the overall network's cross-modal modeling capability.

[0054] Furthermore, the spatial gating graph satisfies the following relationship:

[0055]

[0056] in, This indicates a feature concatenation operation. and These represent the convolutional mappings used to generate the corresponding spatial gating graphs. This represents the Sigmoid activation function.

[0057] Furthermore, the detail enhancement component, the smoothing context component, and the updated bimodal features satisfy the following relationship:

[0058]

[0059]

[0060]

[0061] in, This represents the detail enhancement component extracted from panchromatic features. This represents the smoothed contextual component extracted from multispectral features. This represents the updated multispectral features. This represents the updated panchromatic features. This indicates the fusion jump connection feature. This represents a 3×3 average pooling operation. This indicates an element-wise weighted operation. This indicates a channel compression mapping operation.

[0062] Specifically, step S4 mainly targets large-format inputs. It adopts an asynchronous storage-aware sequential processing method within the network to perform unified forward computation on the divided partitioned regions or labels. This allows different regions to complete feature extraction, cross-modal interaction, feature update, and hierarchical aggregation within the same network computation path. This reduces peak memory usage without disrupting the overall network hierarchical interaction logic and effectively avoids boundary fragmentation, local discontinuity, and stitching artifacts caused by traditional external block inference and post-stitching.

[0063] In some embodiments, step S4 specifically includes: Step S401: The encoder sequentially processes the corresponding partition regions or markers of the large-format input and completes the corresponding feature extraction and update, so that the local regions participate in the multi-stage encoding and cross-modal interaction process in sequence under controlled memory overhead, thereby alleviating the memory pressure when directly reasoning the whole image with ultra-large-format input.

[0064] Step S402: Within the decoder, the partitioned regions or tags are sequentially aggregated and restored step by step, so that the decoding features of different regions are reconstructed at multiple scales under the forward path of the unified network, thereby ensuring that the restoration process is still within the same level of reconstruction framework.

[0065] Step S403: Feature updates and aggregation between different regions are completed within the forward path of the unified network, instead of performing image stitching after independent block inference outside the network. This avoids the problems of cross-block information fragmentation, local structural abrupt changes and boundary discontinuities caused by traditional external block cutting methods.

[0066] Step S404: Reduce boundary stitching artifacts by using asynchronous storage-sensing sequential processing, while maintaining the continuity of large scene context, cross-regional structural consistency, and hierarchical feature interaction capabilities, thereby improving the practical deployment feasibility of the method on large-format remote sensing images such as 4K and 8K.

[0067] Furthermore, the asynchronous storage-aware sequential processing method refers to performing batch feature calculation and aggregation at the internal region level or tag level on large-format inputs without changing the overall network structure and interaction logic, so that all partitions are still under the same level of encoding, gating interaction and decoding recovery framework to complete unified inference.

[0068] Step S5 mainly involves inputting the fusion jump-connected features generated in each stage into the decoder, and performing coarse-to-fine decoding and reconstruction in order from deep to shallow. In each decoding stage, the spatial details and spectral information are restored step by step by combining the fusion jump-connected features of the corresponding scale, and finally outputting high-resolution multispectral residuals. High-resolution multispectral image HRMS output is obtained through residual reconstruction, thereby improving the spatial resolution and enhancing the detail quality of the original multispectral image.

[0069] In some embodiments, step S5 specifically includes: Step S501: Use the deepest fused jump connection features as the initial input of the decoder, so that the deep global semantic information first participates in the high-resolution multispectral reconstruction process, thereby providing a stable high-level semantic foundation for subsequent step-by-step recovery.

[0070] Step S502: In each decoding stage, the current decoding feature is concatenated with the fused skip-connected feature of the corresponding scale to obtain the concatenated feature, so as to combine the decoding information of the current stage with the multi-scale bimodal interaction information retained in the encoding stage, thereby enhancing the detail recovery capability in the decoding process.

[0071] Step S503: Perform upsampling and feature recovery on the stitched features to reconstruct the spatial resolution step by step, and gradually restore the spatial texture details and spectral structure information at each scale, so that high-level semantic information and shallow local detail information are gradually integrated during the decoding process.

[0072] Step S504: Finally, the high-resolution multispectral residual R is output through the convolutional layer, so that the network can focus on learning the detailed information that the upsampled multispectral image needs to compensate for relative to the target high-resolution multispectral image, without having to reconstruct the complete output from scratch.

[0073] Step S505: Perform residual reconstruction on the high-resolution multispectral residual R and the upsampled multispectral image to obtain a full-color sharpened output, thereby enhancing spatial details while maintaining the original multispectral radiative information and improving the spectral consistency and training stability of the final result.

[0074] Furthermore, the decoding process can be represented as:

[0075]

[0076] in, This represents the initial input features of the decoder. This indicates the deepest fusion jump connection feature. This represents the current feature in the k-th decoding stage. This represents the upsampling and feature recovery operations in the k-th decoding stage.

[0077] Furthermore, the final full-color sharpening output satisfies the following relationship:

[0078] in, This indicates a high-resolution multispectral output image. This indicates an upsampled multispectral image. This represents the high-resolution multispectral residual output by the decoder. The residual reconstruction method can focus on learning the spatial detail information that needs to be enhanced while preserving the original spectral information of the upsampled multispectral image, thereby improving training stability and enhancing the spectral consistency of the output results.

[0079] The visual effects of the high-fidelity panchromatic sharpening method for large-format remote sensing imagery provided in this application will be explained below with reference to the accompanying drawings. Figure 4a and Figure 4b The qualitative comparison results of different methods under different dataset downresolution evaluations are shown. Figure 4a For the fusion result, Figure 4b The corresponding residual image is shown. The method in this application outperforms the comparative methods in both overall structure preservation and local detail recovery. The residual image shows fewer high-response areas, smoother edge transitions, and clearer textures and feature outlines in the magnified areas, which are closer to the real reference image, indicating that the proposed method has better spatial detail reconstruction capabilities and spectral preservation performance.

[0080] Figure 5a and Figure 5b The qualitative comparison results of different methods under full-resolution evaluation on different datasets are shown. Figure 5a The result of the fusion; Figure 5b As shown in the corresponding HQNR image, the proposed method outperforms the comparative methods in both overall visual effect and local region reconstruction quality. The high-response areas in the HQNR image are larger and more continuously distributed, while the degraded areas are significantly fewer. This indicates that the proposed method can better maintain spectral fidelity while preserving spatial details, and exhibits superior fusion quality in full-resolution scenes.

[0081] Figure 6 The results show a qualitative comparison of different methods in 8K large-format scenarios with different datasets. From Figure 6 As can be seen, the method of this application is superior to the comparative methods in terms of overall visual consistency and local area reconstruction quality. The building edges, road textures and mountain structures in the magnified area are clearer and more natural, the cross-regional transitions are more continuous, and there are fewer boundary artifacts and local distortions. This shows that the proposed method has better detail recovery ability and large scene structure preservation ability in ultra-large format scenes.

[0082] Secondly, this application also provides a high-fidelity panchromatic sharpening system for large-format remote sensing images. This system implements the high-fidelity panchromatic sharpening method for large-format remote sensing images described in the above embodiments. The system includes: an upsampling module, a feature extraction module, a gated cross-modal interaction module, and a reconstruction module connected sequentially. The upsampling module acquires a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, upsamples the LRMS, and obtains an upsampled multispectral image with the same spatial size as the PAN. The feature extraction module inputs the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch, respectively, to perform multi-stage hierarchical feature extraction, obtaining multispectral and panchromatic features at corresponding scales for each stage. The gated cross-modal interaction module performs staged gated bidirectional exchange between coding stages, reconstructing the multispectral image. The system performs gated cross-modal interaction on spectral and panchromatic features, and generates fused skip-connected features for the decoder while completing the cross-modal interaction. The reconstruction module inputs the fused skip-connected features obtained after staged gated bidirectional exchange into the decoder, and performs coarse-to-fine decoding and reconstruction in order from deep to shallow. At each decoding stage, the fused skip-connected features of the corresponding scale are combined to restore spatial details and spectral information, and finally outputs high-resolution multispectral residual R. The residual reconstruction is performed on R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output. The system adopts an asynchronous storage-aware sequential processing method for large-format input within the network, and performs unified forward computation on the divided partition regions or labels, so that different regions complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction.

[0083] See Figure 7 Another embodiment of this application provides an electronic device, including: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions executable by the at least one processor 110, the instructions being executed by the at least one processor 110 to enable the at least one processor 110 to perform any of the above method embodiments.

[0084] The memory 111 and processor 110 are connected via a bus, which may include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 110 and memory 111. The bus may also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 110 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 110.

[0085] Processor 110 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 111 can be used to store data used by processor 110 during operation.

[0086] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0087] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0088] Based on the above technical solutions, this application provides a high-fidelity panchromatic sharpening method and system for large-format remote sensing images. The method includes the following steps: Step S1: Acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same area, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN; Step S2: Input the upsampled multispectral image and the PAN into the multispectral coding branch and the panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral features and panchromatic features at the corresponding scales of each stage; Step S3: Perform stage-gated bidirectional exchange between each coding stage, perform gated cross-modal interaction on the multispectral features and panchromatic features, and generate a generator for the decoder to use while completing the cross-modal interaction. The following steps are used: Step S4: Within the network, an asynchronous storage-aware sequential processing method oriented towards large-format input is adopted to perform unified forward computation on the divided partitioned regions or labels, so that different regions can complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction; Step S5: The fusion jump-connection features obtained after staged gated bidirectional exchange are input into the decoder, and coarse-to-fine decoding and reconstruction are performed in the order from deep to shallow. At each decoding stage, the fusion jump-connection features of the corresponding scale are combined to restore spatial details and spectral information, and finally output high-resolution multispectral residual R. The residual reconstruction is performed between R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output.

[0089] The high-fidelity panchromatic sharpening method for large-format remote sensing images provided in this application can achieve high-fidelity panchromatic sharpening reconstruction in large-format remote sensing image scenarios. The method extracts features from both high-resolution panchromatic and low-resolution multispectral images using a bi-branch hierarchical approach, modeling spatial texture details in the panchromatic image and spectral context information in the multispectral image, respectively. A staged gating bidirectional exchange mechanism is introduced at each encoding stage to achieve controlled injection of panchromatic details into the multispectral branch and feedback enhancement of the multispectral spectral context into the panchromatic branch, thereby improving spatial detail recovery and spectral consistency preservation. For large-format inputs such as 4K and 8K, the method further employs an asynchronous storage-aware sequential processing approach within the network, reducing peak memory usage while maintaining cross-regional hierarchical feature interaction, effectively reducing boundary stitching artifacts and context fragmentation problems caused by traditional external block inference. Through multi-scale fusion skip-connection feature-guided stepwise decoding and residual reconstruction, the spatial clarity, structural continuity, and spectral fidelity of the reconstruction results can be effectively improved. This method is applicable to various scenarios such as high-resolution remote sensing image interpretation, fine mapping, target recognition, change detection, disaster monitoring, and resource surveys, and has strong engineering practicality and promotional value.

[0090] Those skilled in the art will understand that the above-described embodiments are specific examples of implementing this application, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of this application. Any person skilled in the art can make their own modifications and alterations without departing from the spirit and scope of this application; therefore, the scope of protection of this application should be determined by the scope defined in the claims.

Claims

1. A high fidelity panchromatic sharpening method for large format remote sensing imagery, characterized in that, Includes the following steps: Step S1: Acquire a high-resolution panchromatic image PAN and a low-resolution multispectral image LRMS of the same region, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as PAN; Step S2: Input the upsampled multispectral image and PAN into the multispectral coding branch and panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral features and panchromatic features at each stage corresponding to the scale. Step S3: Perform staged gated bidirectional exchange between each encoding stage, perform gated cross-modal interaction on multispectral and panchromatic features, and generate fused skip-connect features for the decoder to use while completing the cross-modal interaction; Step S4: Within the network, an asynchronous storage-aware sequential processing method oriented towards large-format input is adopted to perform unified forward computation on the partitioned regions or labels, so that different regions can complete feature extraction, cross-modal interaction, feature update and hierarchical aggregation within the same network computation path, thereby reducing peak memory usage and maintaining cross-regional hierarchical feature interaction. Step S5: Input the fused skip-connection features obtained after staged gating bidirectional exchange into the decoder, and perform coarse-to-fine decoding and reconstruction in the order from deep to shallow. In each decoding stage, combine the fused skip-connection features of the corresponding scale to recover spatial details and spectral information, and finally output high-resolution multispectral residual R. Then, perform residual reconstruction with R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output.

2. The method for high-fidelity panchromatic sharpening of large-format remote sensing imagery according to claim 1, wherein, Step S1 specifically includes: Step S101: Acquire the high-resolution panchromatic image PAN and the corresponding low-resolution multispectral image LRMS of the same region respectively; Step S102: Upsample the LRMS using the "23-tap polynomial" interpolation method so that the upsampled multispectral image is consistent with the PAN in spatial size. Step S103: Without changing the original multispectral band information, map LRMS to the spatial resolution corresponding to PAN, thereby providing a unified input for subsequent cross-modal feature extraction and panchromatic sharpening reconstruction.

3. The method for high-fidelity panchromatic sharpening of large-format remote sensing imagery according to claim 1, wherein, Step S2 specifically includes: Step S201: Input the upsampled multispectral image into the multispectral coding branch and input the PAN into the panchromatic coding branch; Step S202: Using a feature extraction backbone network with the same hierarchical structure but different numbers of input channels, multi-stage hierarchical feature extraction is performed on the upsampled multispectral image and PAN respectively; Step S203: Extract contextual information and multi-scale semantic representation with spectral consistency using the multispectral coding branch, and extract spatial texture detail information and structural edge information using the panchromatic coding branch; Step S204: The multispectral coding branch and the panchromatic coding branch extract features in order from shallow to deep. Each stage includes a feature transformation module, so that the multispectral features and panchromatic features gradually expand the receptive field and improve the semantic abstraction ability as the network depth increases. Step S205: After each stage is completed, output the bimodal features at the corresponding scale. The multispectral and panchromatic features updated by the gated bidirectional exchange in the current stage are then input into the next stage for subsequent staged gated bidirectional exchange and decoding reconstruction, so that cross-modal interaction can continue to influence the deeper representation learning process.

4. The method for high-fidelity panchromatic sharpening of large-format remote sensing imagery according to claim 3, wherein, In the s-th encoding stage, the multispectral features and panchromatic features are represented as follows: wherein, denotes the upsampled multispectral image, denotes the feature map of the multispectral encoding branch at stage s, denotes the feature map of the panchromatic encoding branch at stage s, denotes the multispectral feature output by stage s, denotes the panchromatic feature output by stage s.

5. The high fidelity panchromatic sharpening method for large format remote sensing imagery of claim 1 or 3, wherein, The staged gating bidirectional switching in step S3 includes: Step S301, the same stage of multi-spectral features and panchromatic features Splicing in the channel dimension forms a joint representation feature; In step S302, two single-channel spatial gating maps are respectively predicted based on the spliced features through 1x1 convolution and a Sigmoid function and ; wherein the spatial gating map is used to represent the cross-modal information exchange strength at the current spatial position; Step S303, extracting a detail enhancement component reflecting spatial edge and texture change by performing local average smoothing and difference on the full-color feature and according to the spatial gating map adaptively injecting the detail enhancement component into the multi-spectral feature to obtain an updated multi-spectral feature; Step S304, a smooth context component reflecting spectral consistency and regional context relationship is extracted by performing a local average operation on the multi-spectral feature and according to the spatial gating map The smooth context component is fed back into the panchromatic feature to obtain an updated panchromatic feature Step S305: Concatenate the updated multispectral features and the updated panchromatic features along the channel dimension, and compress them to the preset channel dimension through a 1×1 convolution to obtain the fused skip-connected features. ; Step S306: The updated multispectral features and updated panchromatic features are then passed to subsequent coding stages to achieve the step-by-step propagation of cross-modal information in the multi-stage hierarchical network.

6. The high-fidelity panchromatic sharpening method for large-format remote sensing images according to claim 1, characterized in that, Step S4 specifically includes: Step S401: The encoder sequentially processes the corresponding partition regions or markers of the large-format input and completes the corresponding feature extraction and update. Step S402: Perform sequential aggregation and step-by-step recovery of the partitioned regions or tags within the decoder; Step S403: Complete feature updating and aggregation between different regions within the forward path of the unified network, rather than performing block inference independently outside the network and then stitching the image together. Step S404: Reduce boundary stitching artifacts and maintain the continuity of the large scene context through the asynchronous storage-aware sequential processing method.

7. The high-fidelity panchromatic sharpening method for large-format remote sensing images according to claim 1, characterized in that, Step S5 specifically includes: Step S501: Use the deepest layer fused skip connection features as the initial input to the decoder; Step S502: In each decoding stage, the current decoded feature is concatenated with the fused skip-connected feature of the corresponding scale to obtain the concatenated feature; Step S503: Perform upsampling and feature restoration on the stitched features to reconstruct the spatial resolution step by step; Step S504: Finally, high-resolution multispectral residual R is output through the convolutional layer; Step S505: Perform residual reconstruction on the high-resolution multispectral residual R and the upsampled multispectral image to obtain full-color sharpened output.

8. A high-fidelity panchromatic sharpening system for large-format remote sensing imagery, the system being used to implement the high-fidelity panchromatic sharpening method for large-format remote sensing imagery as described in any one of claims 1 to 7, characterized in that, The system includes: an upsampling module, a feature extraction module, a gated cross-modal interaction module, and a reconstruction module, connected in sequence; among them, The upsampling module is used to acquire a high-resolution panchromatic image (PAN) and a low-resolution multispectral image (LRMS) of the same region, and upsample the LRMS to obtain an upsampled multispectral image with the same spatial size as the PAN. The feature extraction module is used to input the upsampled multispectral image and PAN into the multispectral coding branch and the panchromatic coding branch respectively, and perform multi-stage hierarchical feature extraction to obtain multispectral features and panchromatic features at each stage corresponding to the scale. The gated cross-modal interaction module is used to perform staged gated bidirectional exchange between each encoding stage, perform gated cross-modal interaction on multispectral features and panchromatic features, and generate fused skip-connection features for the decoder to use while completing the cross-modal interaction; The reconstruction module is used to input the fused skip-connection features obtained after staged gated bidirectional exchange into the decoder, and perform coarse-to-fine decoding and reconstruction in the order from deep to shallow. At each decoding stage, the fused skip-connection features of the corresponding scale are combined to recover spatial details and spectral information, and finally output high-resolution multispectral residual R. The residual reconstruction is performed between R and the upsampled multispectral image to obtain the high-resolution multispectral image HRMS output. The system employs an asynchronous storage-aware sequential processing method oriented towards large-format input within the network. It performs unified forward computation on the divided partitioned regions or labels, enabling different regions to complete feature extraction, cross-modal interaction, feature update, and hierarchical aggregation within the same network computation path. This reduces peak memory usage while maintaining cross-regional hierarchical feature interaction.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the high-fidelity panchromatic sharpening method for large-format remote sensing imagery as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the high-fidelity panchromatic sharpening method for large-format remote sensing images as described in any one of claims 1 to 7.

Images