Self-Supervised Hyperspectral and Spatial Remote Sensing Image Fusion Method and Device Based on Prior Image Constraint

Abstract

A self-supervised remote sensing image spatial-spectral fusion method and device based on prior image constraint, which relates to the technical field of remote sensing satellites. Among them, the method includes: obtaining a panchromatic image and a multispectral image; generating a prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, where the prior image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network for the panchromatic image and the multispectral image; inputting the panchromatic image, the multispectral image, and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fusion image output by the remote sensing image spatial-spectral fusion network; implementing the technical solution provided by this application enhances the generalization ability of the model when processing data from different sensors and achieves a more stable fusion effect.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing satellites, and particularly to a self-supervised remote sensing image spatial-spectral fusion method and device based on prior image constraints. Background Art

[0002] Remote sensing satellites can usually acquire panchromatic images and multispectral images simultaneously. Among them, panchromatic images have high spatial resolution but only contain single-band information, while multispectral images have rich spectral information but low spatial resolution. In order to obtain remote sensing images with both high spatial resolution and rich spectral information, it is necessary to perform spatial-spectral fusion processing on panchromatic images and multispectral images. Spatial-spectral fusion technology has important application values in fields such as agricultural monitoring, urban planning, and environmental monitoring.

[0003] Currently, deep learning-based spatial-spectral fusion methods mainly adopt the way of supervised learning. By constructing a simulated downscaled dataset to obtain training samples, taking the original multispectral image as the reference image label, and performing supervised training on the fusion network. Such methods usually use the downscaled panchromatic image and multispectral image as network inputs, and optimize network parameters by minimizing the error between the fusion result and the reference image. In addition, some studies have also proposed using unsupervised learning methods such as generative adversarial networks to directly train with images of original resolution.

[0004] However, existing deep learning methods often lack sufficient consideration of the inherent relationship between panchromatic images and multispectral images when performing spatial-spectral fusion. Due to differences in spectral response and spatial resolution between images collected by different sensors, it is difficult to accurately capture this relationship only relying on the feature extraction ability of the network itself, which easily leads to distortion in the preservation of spatial details and spectral information in the fusion result. Especially when processing data from different sensors, the generalization ability of the model is poor and it is unable to maintain a stable fusion effect. Summary of the Invention

[0005] The present application provides a self-supervised remote sensing image spatial-spectral fusion method and device based on prior image constraints, which enhances the generalization ability of the model when processing data from different sensors and achieves a more stable fusion effect.

[0006] In the first aspect of the present application, a self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints is provided, including:

[0007] Obtain a panchromatic image and a multispectral image;

[0008] Generate a prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, where the prior image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network for the panchromatic image and the multispectral image;

[0009] Input the panchromatic image, the multispectral image, and the prior image into the remote sensing image spatio-spectral fusion network to obtain the target fusion image output by the remote sensing image spatio-spectral fusion network;

[0010] Among them, the remote sensing image spatio-spectral fusion network is trained in a self-supervised manner based on a target loss function, and the target loss function is constructed by calculating the probability distribution differences of the spatial loss and the spectral loss among the target fusion image, the sample panchromatic image, and the sample multispectral image.

[0011] In the second aspect of the present application, a self-supervised remote sensing image spatio-spectral fusion device based on prior image constraint is provided, including:

[0012] An initial image acquisition module, configured to acquire a panchromatic image and a multispectral image;

[0013] A prior image generation module, configured to generate a prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, and the prior image is used to constrain the fusion process of the remote sensing image spatio-spectral fusion network for the panchromatic image and the multispectral image;

[0014] A target image fusion module, configured to input the panchromatic image, the multispectral image, and the prior image into the remote sensing image spatio-spectral fusion network to obtain the target fusion image output by the remote sensing image spatio-spectral fusion network;

[0015] Among them, the remote sensing image spatio-spectral fusion network is trained in a self-supervised manner based on a target loss function, and the target loss function is constructed by calculating the probability distribution differences of the spatial loss and the spectral loss among the target fusion image, the sample panchromatic image, and the sample multispectral image.

[0016] In the third aspect of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, it implements the self-supervised remote sensing image spatio-spectral fusion method according to any one of the above.

[0017] In the fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the self-supervised remote sensing image spatio-spectral fusion method according to any one of the above.

[0018] In summary, one or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0019] By introducing a prior image as a constraint, which contains the spatial information of the panchromatic image and the spectral information of the multispectral image, prior knowledge of the inherent relationship between images is provided for the remote sensing image spatial-spectral fusion network. At the same time, the prior image, the panchromatic image, and the multispectral image are jointly input into the fusion network, and a self-supervised training method is adopted. The target loss function is used to calculate the probability distribution differences of the spatial loss and the spectral loss between the target fusion image, the sample panchromatic image, and the sample multispectral image, so that the network can fully learn and utilize the inherent relationship between images during the training process. This self-supervised learning method based on prior constraints avoids the limitations of relying solely on the network's own feature extraction ability, improves the accuracy of the fusion result in maintaining spatial details and spectral information, enhances the generalization ability of the model when processing data from different sensors, and achieves a more stable fusion effect. Brief Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a schematic flowchart of a self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint provided by an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of the framework of a remote sensing image spatial-spectral fusion network provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the structure of a residual module provided by an embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the structure of a dual attention module provided by an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of a dense fusion module provided by an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of the construction of a loss function provided by an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the process of a gray degradation network provided by an embodiment of the present application;

[0028] Figure 8 It is a schematic diagram of the structure of a self-supervised remote sensing image spatial-spectral fusion device based on prior image constraint provided by an embodiment of the present application;

[0029] Figure 9 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0030] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below with reference to the accompanying drawings in the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without making creative efforts shall fall within the protection scope of the present application.

[0031] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of a self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints provided by an embodiment of the present application. This method can be implemented depending on a computer program, can be implemented depending on a single-chip microcomputer, or can run on a self-supervised remote sensing image spatial-spectral fusion system based on the von Neumann architecture. This computer program can be integrated into an application or run as an independent tool-type application. Specifically, this method may include the following steps:

[0032] S101. Obtain a panchromatic image and a multispectral image.

[0033] Among them, the panchromatic image refers to a single-band image obtained by a sensor within a relatively wide band range. The panchromatic image is used to provide the spatial structure information of ground objects, including high-frequency detail information such as texture features and edge contours.

[0034] Correspondingly, the multispectral image refers to a multi-band image obtained by a sensor simultaneously within multiple independent narrow band ranges. The multispectral image is used to provide the spectral reflection characteristics of ground objects, which helps to distinguish the physical and chemical properties of different ground objects and achieve ground object classification and recognition.

[0035] Specifically, before performing remote sensing image spatial-spectral fusion, it is first necessary to obtain the panchromatic image and the multispectral image to be fused. Due to the physical limitations of satellite sensors, the optical remote sensing images obtained by the same satellite usually only have advantages in terms of spatial resolution or spectral resolution in one aspect: the panchromatic image has a high spatial resolution but limited spectral information, and the multispectral image has rich spectral information but a low spatial resolution. Therefore, it is necessary to fuse these two complementary image information to obtain a fused image that has both high spatial resolution and rich spectral information.

[0036] S102. Generate a prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image. The prior image is used to constrain the fusion process of the remote sensing image pan-spectral fusion network for the panchromatic image and the multispectral image.

[0037] Among them, the prior image refers to a reference image generated by reasonably combining the spatial information of the panchromatic image and the spectral information of the multispectral image. The prior image is used to provide training reference and constraint guidance for the remote sensing image pan-spectral fusion network, helping the network to better balance the preservation of spatial detail information and spectral feature information during the self-supervised learning process, thereby improving the quality of the fusion result.

[0038] Specifically, before fusing the panchromatic image and the multispectral image, it is necessary to first generate a prior image. Due to the lack of real high-resolution multispectral images as training references, traditional supervised learning methods often require manual labeling or simulating degraded datasets, which not only increases the workload but also may cause loss of spatial information. Therefore, this application proposes to use the prior image to constrain the fusion process, and generate a prior image with reference value by reasonably combining the spatial information of the panchromatic image and the spectral information of the multispectral image.

[0039] Based on the above embodiments, as an alternative embodiment, the spatial information in the panchromatic image can be injected into the multispectral image to obtain a prior image.

[0040] Specifically, although the panchromatic image has only a single band, its spectral response range is relatively wide, containing the comprehensive reflection characteristics of ground objects in multiple bands. Therefore, the spatial information contained in the panchromatic image can be migrated to the multispectral image by establishing the statistical relationship between the panchromatic image and each band of the multispectral image.

[0041] In a feasible implementation manner, the Gram-Schmidt (GS) method can be adopted. This method first performs mathematical orthogonalization on the multispectral data, replaces the first orthogonal basis with the panchromatic image, and then performs an inverse transformation to generate a prior image. The above method can make full use of the spatial information in the panchromatic image on the basis of maintaining the original spectral characteristics of the multispectral image, and improve the spectral fidelity of the fusion result.

[0042] In another feasible implementation, the Nearest Neighbor Diffusion (NND) method can be adopted. This method first separates the high-frequency and low-frequency information of the panchromatic image through low-pass filtering, then calculates the similarity weights between the panchromatic image and each band of the multispectral image, and finally adaptively injects the high-frequency information of the panchromatic image into the multispectral image based on these weights. The above method can not only effectively preserve the spectral features in the multispectral image, but also adaptively adjust the degree of information injection according to the correlation between different bands and the panchromatic image, avoiding problems such as over-enhancement or under-enhancement.

[0043] For generating the prior image using the Gram-Schmidt (GS) method. The GS method aligns the spectral characteristics of the panchromatic image and the multispectral image through an orthogonalization process, minimizing spectral distortion while enhancing the spatial resolution, thus obtaining a better fusion effect.

[0044] Specifically, first, the multispectral image is upsampled to make its spatial resolution consistent with that of the panchromatic image. Then, the upsampled multispectral image and the panchromatic image are stitched in the spectral dimension to form a sequence of images to be processed. Next, the GS orthogonalization transform is performed on this sequence of images to obtain a set of orthogonal bases. In the orthogonal bases, the first orthogonal basis is replaced with the panchromatic image, while the other orthogonal bases remain unchanged. Finally, the inverse GS transform is performed on the replaced orthogonal bases to obtain the prior image.

[0045] Exemplarily, the Gram-Schmidt spatial-spectral fusion method can be simply expressed by the following formula:

[0046] In the formula, cat(a,b) represents stitching a and b along the spectral bands; represents replacing the first orthogonal basis. The GS method performs mathematical orthogonalization on the MS data and executes the inverse transform. Specifically, this method first orthogonalizes the MS data, replaces the first orthogonal basis with the PAN image, and then performs the inverse transform to generate the prior image.

[0047] The above spatial information injection method based on the GS transform can effectively preserve the spectral characteristics of the multispectral image, while reasonably distributing the spatial detail information of the panchromatic image into each spectral band. Through the orthogonalization process, the GS method reduces the correlation between bands and spectral distortion, making the generated prior image have better spectral fidelity. This provides a reliable basis for subsequent spatial information injection and helps improve the quality of the fusion result.

[0048] For generating a prior image using the Nearest Neighbor Diffusion (NND) method. The NND method utilizes the principles of nearest neighbor search and local similarity, and can effectively transfer the high-frequency information in the panchromatic image to the multispectral image while keeping the spectral features from undergoing obvious distortion.

[0049] Specifically, first, the multispectral image is upsampled to make its spatial resolution consistent with that of the panchromatic image. Then, high-frequency detail information, including texture features and edge contours, etc., is extracted from the panchromatic image. Next, for each band of the upsampled multispectral image, the most similar image patch is searched in the panchromatic image to establish a corresponding relationship. Based on this corresponding relationship, through the way of local mean diffusion, the high-frequency information in the panchromatic image is gradually injected into each band of the multispectral image, and finally the prior image is obtained.

[0050] Exemplarily, the NND algorithm utilizes pixel-level similarity and the diffusion process to effectively spread spatial details in the image. The algorithm extracts high-frequency information from the P image and the MS image respectively, and its implementation process is as shown in the following formula:

[0051]

[0052]

[0053] Where, LPF(·) represents the low-pass filtering operation function. represents the i-th band of 、 and represent the low-frequency information of the P image, the high-frequency information of the P image, and the high-frequency information of respectively. Therefore, the weights

[0054]

[0055] of P and MS at each pixel position (x, y) and band i can be calculated as follows: Where, Z(x, y) is the normalization constant such that . The parameter

[0056] controls the attenuation degree of the weight. Subsequently, the high-frequency details of the P image are injected into the multispectral bands through the weight

[0057]

[0058] In the formula, C represents the number of bands of MS, Indicates the final fusion result. The NND method utilizes the spatial information of the PAN image to guide the enhancement of the MS data. Specifically, it calculates the similarity between the PAN image and each MS band in the low-frequency domain, assigns adaptive weights based on this similarity, and then injects the high-frequency details of the PAN image into the MS data. The resulting prior image retains both the spatial details of the PAN image and the spectral fidelity of the MS bands.

[0059] Based on the above embodiments, after generating the prior image through the GS and NND methods, considering that the ground object targets in remote sensing images often have multi-scale characteristics, extracting features only at a single scale may lose some important spatial and spectral information. Therefore, multi-scale feature extraction can be performed to fully capture the image details at different scales, thereby enhancing the integrity and accuracy of the fusion effect.

[0060] Specifically, the multi-scale feature extraction process first performs multiple downsampling operations on the panchromatic image, multispectral image, and prior image. In this application, average pooling is used to implement downsampling because average pooling can maintain the main structural information of the image while reducing the image resolution, avoiding sudden loss of information or drastic changes in features. Through average pooling, each larger-scale feature map is converted into a smaller-scale feature map, forming a multi-scale feature pyramid structure.

[0061] At each scale level, the network separately performs feature extraction and fusion operations on the downsampled panchromatic image, multispectral image, and prior image. This hierarchical processing method enables the network to simultaneously focus on image features at different scales: larger-scale feature maps mainly contain the global structural information of the image, while smaller-scale feature maps retain more local detail information. By processing in parallel at different scales, the network can learn a more rich and complete image feature representation.

[0062] After completing the extraction and fusion of features at each scale, it is necessary to resample the feature maps at different scales to the same spatial resolution. In this application, bicubic interpolation is used to implement the upsampling operation because, compared with nearest neighbor interpolation and bilinear interpolation, bicubic interpolation can produce a smoother interpolation result, reducing the jagged effect and distortion phenomenon during the resampling process. Through bicubic interpolation, the smaller-scale feature maps are resampled to a larger scale, maintaining the continuity and smoothness of the features.

[0063] S103. Input the panchromatic image, multispectral image, and prior image into the remote sensing image spatio-spectral fusion network to obtain the target fusion image output by the remote sensing image spatio-spectral fusion network. The remote sensing image spatio-spectral fusion network is trained in a self-supervised manner based on the target loss function, and the target loss function is constructed by calculating the probability distribution differences of the spatial loss and spectral loss among the target fusion image, sample panchromatic image, and sample multispectral image.

[0064] Among them, the remote sensing image spatio-spectral fusion network refers to a neural network structure based on deep learning. This network can simultaneously process three-way inputs of panchromatic image, multispectral image, and prior image, and through the collaborative action of multiple functional modules, achieve the fusion of high spatial resolution and high spectral resolution images.

[0065] In the embodiments of this application, it can be understood as a three-stream fusion network architecture composed of residual modules, dual attention modules, and dense fusion modules. Among them, the residual module is responsible for feature extraction and mapping, and enhances the feature learning ability of the network through the residual connection structure; the dual attention module includes two sub-modules of spatial attention and spectral attention, which respectively focus on the reconstruction of spatial details and spectral features; the dense fusion module effectively fuses multi-scale features through dense connection methods.

[0066] Furthermore, the remote sensing image spatio-spectral fusion network is used to fuse the multispectral image with low spatial resolution but rich spectral information and the panchromatic image with high spatial resolution but limited spectral information. At the same time, it uses the prior image to provide additional constraint information, and finally generates the target fusion image that not only maintains high spatial resolution but also has rich spectral information. This network is trained through self-supervised learning, without the need for artificially made training labels, and can better maintain the feature information of the original data and improve the quality of the fusion result.

[0067] Specifically, after obtaining the prior image, it is necessary to input the panchromatic image, multispectral image, and prior image into the remote sensing image spatio-spectral fusion network to obtain the target fusion image. Due to the lack of real high-resolution multispectral images as training labels, traditional supervised learning methods often need to train the network by artificially making labels or simulating degraded data sets, which not only increases the workload but also may introduce additional errors. To solve this problem, the embodiments of this application adopt a self-supervised learning method to train the remote sensing image spatio-spectral fusion network, and directly use the input panchromatic image, multispectral image, and prior image as training references.

[0068] Exemplarily, please refer to Figure 2 , Figure 2 which shows a schematic diagram of the remote sensing image spatio-spectral fusion network framework provided by the embodiments of this application.

[0069] Specifically, in the remote sensing image spatial-spectral fusion network architecture, first, the MS image is upsampled to generate an upsampled multispectral image (UPMS). To better adapt to different sensor data, a traditional spatial-spectral fusion method is used to fuse the UPMS with the PAN image to obtain a prior image. This prior image, together with the PAN and MS, enters the network as three-stream inputs. To more effectively extract deep features, these three-stream input data are downsampled multiple times through an average pooling layer (AveragePooling, AP) to generate multi-scale input images. At each scale, the data sequentially passes through an encoder, a multi-residual feature extraction module (MRFEM), a dual attention mechanism module (DAM), and a densely connected fusion module (DenseFuse) to obtain the target fusion features at that scale. Finally, these target fusion features are resampled to the same scale through bicubic interpolation and stitched together, and after being processed by a decoder, a high-resolution multispectral image is generated. Finally, a self-supervised learning method is adopted, combined with spatial, spectral, and spatial-spectral joint loss functions to guide the training of the network, ensuring better feature fusion in both spatial and spectral dimensions.

[0070] Exemplarily, the SPTN model can be expressed as:

[0071]

[0072] In the formula, f(.) represents the SPTN network, represents the values related to different parameter settings in the network. 、 and serve as the inputs to the network.

[0073] The remote sensing image spatial-spectral fusion network utilizes a number of advanced architecture features to ensure optimal feature extraction and fusion quality. The key includes: a multi-scale module for capturing features at different resolutions through sensors, ensuring that the model can identify details at various scales; a residual module to enhance the depth and efficiency of learning, enabling the model to extract more complex features; a dual attention module to selectively focus on spatial and spectral features, thus more effectively balancing the preservation of spatial resolution and spectral information; and a dense fusion module to achieve full fusion of features through dense connections within the network, ensuring rich integration of features at different levels and improving the accuracy and integrity of the fusion effect.

[0074] The above-mentioned network training method based on self-supervised learning avoids the cumbersome process of manually creating training labels, directly uses the input image as the supervision signal, and better preserves the characteristics of the original data. At the same time, the multi-module cascaded network architecture and the multi-dimensional loss function design enable the network to fully learn the balance relationship between spatial details and spectral features, effectively improving the quality of the fused image. In particular, the introduction of the dual attention mechanism and the dense connection structure enhances the network's perception ability of important features and improves the effect of feature extraction and fusion.

[0075] Based on the above embodiments, as an optional embodiment, the remote sensing image spatial-spectral fusion network includes a residual module, a dual attention module, and a dense fusion module; in S103: the step of inputting the panchromatic image, the multi-spectral image, and the prior image into the remote sensing image spatial-spectral fusion network to obtain the target fused image output by the remote sensing image spatial-spectral fusion network may specifically further include the following steps:

[0076] S301. Extract features from the panchromatic image, the multi-spectral image, and the prior image through the residual module to obtain deep features.

[0077] Specifically, in order to effectively extract and fuse the feature information of the input image, it is first necessary to extract features from the panchromatic image, the multi-spectral image, and the prior image through the residual module. Since the three input images have different feature distributions and data structures, directly performing feature fusion may result in information loss or distortion. To solve this problem, the embodiment of the present application designs a residual module including an encoder, a convolutional layer, a normalization layer, and an activation function layer, and realizes the extraction of different image features through feature mapping and residual learning.

[0078] Based on the above embodiments, as an optional embodiment, please refer to Figure 3 , Figure 3 which is a schematic structural diagram of a residual module provided by an embodiment of the present application. The residual module includes an encoder, a convolutional layer, a normalization layer, and an activation function layer; in S301: the step of mapping the spectral features in the panchromatic image, the multi-spectral image, and the prior image to spatial features through the residual module to obtain deep features may specifically further include the following steps:

[0079] S401. Encode the panchromatic image, the multi-spectral image, and the prior image respectively through the encoder to obtain encoded features.

[0080] During the pan-spectral fusion of remote sensing images, due to the different data structures and feature distributions of the panchromatic image, the multi-spectral image, and the prior image, directly performing feature extraction and fusion may lead to information loss or feature misalignment. To solve this problem, it is necessary to first map the three input images into a unified feature space. In the embodiments of this application, a dedicated encoder is designed to perform feature encoding on different input images, ensuring that subsequent feature extraction and fusion can be carried out in the same feature domain.

[0081] Specifically, the encoder adopts a multi-layer convolutional network structure to extract and transform features according to the characteristics of different input images. For the panchromatic image, the encoder focuses on extracting the spatial structure information therein, including high-frequency details such as texture features and edge contours; for the multi-spectral image, the encoder emphasizes maintaining its rich spectral information while enhancing its spatial expression ability; for the prior image, the encoder needs to simultaneously pay attention to the information in both the spatial and spectral dimensions to provide effective fusion constraints. Through this targeted design, it is ensured that the important features of various input images can be fully retained during the encoding process.

[0082] Exemplarily, the encoding process can be expressed as:

[0083] E = Encoder(Input);

[0084] In the formula, E represents the encoded features, Input is the input data (such as the panchromatic image, the multi-spectral image, or the prior image), and Encoder(·) represents the encoder function. By applying the encoding process, the encoded representation EP of the panchromatic image, the encoded representation EMS of the multi-spectral image, and the encoded representation EPI of the prior image are respectively obtained. These encoded features have the same dimension and feature expression method, providing a unified basis for subsequent feature fusion.

[0085] The above feature transformation method based on the encoder can effectively extract the key features of different types of images through a specially designed encoding structure; secondly, mapping the features in different domains to a unified feature space solves the basic problem of feature fusion.

[0086] S402. Respectively splice the encoded features corresponding to the panchromatic image, the multi-spectral image, and the prior image in pairs to obtain initial features.

[0087] After obtaining the encoded features of the panchromatic image, the multispectral image, and the prior image, it is necessary to splice these features to make full use of the complementary information between different images. Since the three input images respectively contain different key information: the panchromatic image provides high-resolution spatial details, the multispectral image provides rich spectral information, and the prior image provides fusion reference and constraints, directly splicing the three features at once may lead to mutual interference of information or weakening of features. To better maintain the advantages of various features and achieve information complementarity, the embodiments of this application adopt a pairwise splicing strategy, and gradually extract and splice effective information through combinations between two features.

[0088] Specifically, the encoded features EP, EMS, and EPI are combined in pairs, and the encoded features of the panchromatic image and the multispectral image, the encoded features of the multispectral image and the prior image, and the encoded features of the panchromatic image and the prior image are spliced respectively. The splicing process of each pair of features first performs splicing in the channel dimension, and then realizes the interaction between features and the extraction of information through convolution operations. This pairwise splicing method can more finely control the information flow between different types of features, ensuring that important features will not be overly weakened or lost during the splicing process.

[0089] Exemplarily, the pairwise feature splicing process can be expressed as:

[0090]

[0091] Among them, cat_each(a, b, c) represents pairwise channel splicing of a, b, and c, and then overall splicing. F represents the initial feature obtained after splicing, and MRFEM() represents the operation of using multiple deep residual modules for feature extraction and splicing. This design ensures that sufficient information exchange and feature recombination can occur between each pair of features.

[0092] The above-mentioned feature processing based on pairwise splicing reduces the information interference that may occur when directly fusing three features through combinations of two features; secondly, the splicing of each pair of features can focus on specific information complementary relationships. For example, the splicing of the panchromatic image and the multispectral image focuses on the integration of spatial-spectral information, and the splicing of the multispectral image and the prior image pays more attention to the constraint and optimization of features; finally, the splicing results of multiple pairs of features together constitute a more complete feature representation, providing a rich information basis for subsequent deep feature extraction. Experimental results show that this pairwise splicing strategy can better maintain the dominant features of various images and at the same time achieve effective complementarity between different features.

[0093] S403. Process the initial feature through a convolutional layer, a normalization layer, and an activation function layer to obtain a deep feature.

[0094] After obtaining the initial features, in order to further extract and optimize the feature representation, it is necessary to perform in-depth processing and transformation of the features through a deep network structure. Since the initial features mainly contain basic spatial structures and spectral information, deeper feature extraction is still required to capture more complex feature patterns and hierarchical relationships. To this end, the embodiments of the present application design a processing unit composed of a convolutional layer, a normalization layer, and an activation function layer. Through the collaborative action of these basic components, in-depth extraction and optimization of features are achieved.

[0095] Specifically, first, the convolutional layer is used to extract features in the spatial domain from the initial features. The convolutional kernel captures feature patterns in local regions, including spatial structure information such as textures and edges, by means of a sliding window. Next, the normalization layer is used to perform standardization processing on the features. This step can reduce the internal shift of the feature distribution and improve the training stability and convergence speed of the network. Finally, the activation function layer is introduced to introduce non-linear transformation capabilities, enabling the network to learn more complex feature mapping relationships. In particular, a residual connection structure is introduced in these processing units, adding the input features directly to the output features. This design can not only alleviate the problem of gradient disappearance in the training of deep networks but also retain important information in the original features.

[0096] Exemplarily, the processing process of the deep residual block can be expressed as:

[0097]

[0098] In the formula, F n represents the output feature of the nth layer, ω n is the weight parameter of this layer, F n-1 is the output feature of the previous layer, and bn is the bias parameter. By cascading multiple such residual blocks, the network can gradually learn more abstract and expressive feature representations, and finally obtain deep features containing rich semantic information. Each residual block not only learns the residual part of the features but also retains important information in the original features through skip connections, ensuring the effective transmission of information in the deep network.

[0099] The above-mentioned feature extraction method based on multiple processing units includes the multi-layer stacking of convolutional layers, enabling the network to gradually extract various feature patterns from low-level to high-level; secondly, the introduction of the normalization layer improves the training stability and efficiency of the network; thirdly, the non-linear activation function enhances the feature expression ability of the network.

[0100] S302. Extract the spatial features and spectral features from the deep features through the dual attention module, and splice the spatial features and spectral features to obtain the spliced features.

[0101] After obtaining the depth features, in order to more effectively utilize the spatial and spectral information in the features, a mechanism that can adaptively focus on different feature dimensions is required. Since the core objective of remote sensing image spatial-spectral fusion is to simultaneously maintain high spatial resolution and rich spectral information, directly processing the depth features may neglect some important feature representations. Therefore, the embodiment of this application designs a dual attention module that includes a spatial attention sub-module and a spectral attention sub-module. Through two independent but cooperative attention mechanisms, the expressions of spatial features and spectral features are enhanced respectively, and effective extraction and splicing of features are achieved.

[0102] Based on the above embodiments, as an alternative embodiment, please refer to Figure 4 , Figure 4 which is a schematic structural diagram of a dual attention module provided by the embodiment of this application. The dual attention module includes a spatial attention sub-module and a spectral attention sub-module. In S302: The step of extracting the spatial features and spectral features in the depth features through the dual attention module and splicing the spatial features and spectral features to obtain the spliced features can specifically further include the following steps:

[0103] S501. Perform global max pooling operation and global average pooling operation on the depth features through the spatial attention sub-module, and obtain a spatial attention map through convolution operation. Multiply the spatial attention map with the depth features to obtain spatial features.

[0104] During the process of remote sensing image spatial-spectral fusion, effectively extracting and enhancing spatial features is crucial for maintaining the detail information and structural features of the image. Since the depth features contain rich but unevenly distributed spatial information, direct processing may neglect some key spatial structures. Therefore, the embodiment of this application designs a spatial attention sub-module, which adaptively generates a spatial attention map by simultaneously considering the maximum response and average distribution of the feature map, thereby highlighting the important spatial region features.

[0105] Specifically, the spatial attention sub-module first performs global max pooling and global average pooling operations on the input depth features. Global max pooling can capture the most significant activation responses in the feature map, which helps to locate important spatial structures and edge information; global average pooling provides the overall statistical information of the feature map, reflecting the general feature distribution in different regions. These two complementary pooling operations together construct a more comprehensive description of the spatial features. Subsequently, the results of the two poolings are concatenated in the channel dimension, integrated through a 3×3 convolutional layer, and finally the feature is mapped to between 0 and 1 using the Sigmoid activation function to generate the spatial attention map. The value at each position in the spatial attention map represents the importance of that position in the spatial dimension. Finally, the spatial attention map is multiplied element-wise with the original depth features to enhance the features in the important spatial regions.

[0106] Exemplarily, the calculation process of the spatial attention mechanism can be expressed as:

[0107]

[0108]

[0109] In the formula, GMP() and GAP() represent global max pooling and global average pooling operations respectively. represents the Sigmoid activation function, represents the element-wise multiplication operation. Through the above calculation method, the spatial attention sub-module can generate the attention map S reflecting spatial importance and use it to modulate the original feature F to obtain the enhanced spatial feature F_spa.

[0110] The above feature enhancement mechanism based on spatial attention provides a more comprehensive description of spatial features by combining max pooling and average pooling operations; secondly, the adaptively generated attention map can dynamically adjust the attention area according to the specific situation of the input features; finally, the soft weighting method of the attention mechanism avoids information loss that may be caused by hard threshold division.

[0111] S502. Perform global average pooling operation on the depth features through the spectral attention sub-module, obtain the spectral attention map through the convolutional layer, and multiply the spectral attention map with the depth features to obtain the spectral features.

[0112] In the process of remote sensing image spatial-spectral fusion, maintaining and enhancing spectral features is of great significance for ensuring the spectral fidelity of the fusion result. Since the spectral information in the deep features is distributed in different feature channels and there are differences in the importance of each channel, direct processing may weaken the key spectral features. Therefore, the embodiment of this application designs a spectral attention sub-module, which adaptively generates a spectral attention map by establishing the dependency relationship between feature channels, thereby highlighting the important spectral features.

[0113] Specifically, the spectral attention sub-module first performs a global average pooling operation on the input deep features. Global average pooling can extract the global statistical information of each feature channel and effectively capture the overall response characteristics of different spectral channels. This pooling operation compresses the two-dimensional spatial features into a one-dimensional channel descriptor, enabling the network to better focus on the channel-level feature distribution. Subsequently, the pooled features are sequentially passed through two 1×1 convolutional layers for feature transformation. The first convolutional layer is responsible for dimensionality reduction to reduce the computational amount and extract the inter-channel correlation, and the second convolutional layer restores the features to the original number of channels. Through this dimensionality reduction and restoration operation, the non-linear relationship between channels is effectively modeled. Finally, the Sigmoid activation function is used to map the features to between 0 and 1 to generate the spectral attention map. Each value in the spectral attention map represents the importance degree of the corresponding channel in the spectral dimension. Finally, the spectral attention map is multiplied element-wise with the original deep features to enhance the important spectral features.

[0114] Exemplarily, the calculation process of the spectral attention mechanism can be expressed as:

[0115]

[0116]

[0117] In the formula, F represents the input deep features, GAP() represents the global average pooling operation, fconv represents the concatenation operation of two 1×1 convolutional layers, σ represents the Sigmoid activation function, represents the element-wise multiplication operation. Through this calculation method, the spectral attention sub-module can generate an attention map Q reflecting the channel importance and use it to modulate the original feature F to obtain the enhanced spectral feature F_spe.

[0118] The above feature enhancement mechanism based on spectral attention has significant advantages: First, the global information at the channel level is obtained through global average pooling, effectively characterizing the feature distribution of different spectral channels; Second, the design of two layers of convolution fully models the dependency relationship between channels through dimensionality reduction and restoration operations; Finally, the adaptively generated attention weights can dynamically adjust the importance of different channels according to the input features.

[0119] S503. Concatenate the spatial feature and the spectral feature to obtain a concatenated feature.

[0120] After separately obtaining the enhanced spatial feature and spectral feature, it is necessary to effectively concatenate these two complementary features to obtain a feature representation that simultaneously has high-quality spatial expression and rich spectral information. Since the spatial feature F_spa mainly contains the structural detail information of the image, while the spectral feature F_spe retains rich spectral response features, directly simply adding or concatenating the two features may lead to mutual interference or information redundancy between the features. Therefore, the embodiment of this application designs a feature concatenation strategy to effectively integrate the spatial feature and the spectral feature through the combination of feature concatenation and convolutional transformation.

[0121] Specifically, first concatenate the spatial feature F_spa and the spectral feature F_spe enhanced by the spatial attention sub-module and the spectral attention sub-module respectively in the channel dimension. This concatenation operation retains the complete information of the two features, enabling subsequent processing to make full use of the complementary relationship between the two features. Subsequently, process the concatenated feature through a 5×5 convolutional layer. The larger convolutional kernel size helps to capture a larger range of feature dependencies and achieve deep fusion between the features. This convolutional operation can not only learn the non-linear mapping relationship between the spatial feature and the spectral feature, but also reduce the feature dimension, remove redundant information, and generate a more compact and effective feature representation.

[0122] Exemplarily, the feature fusion process can be expressed as:

[0123]

[0124] In the formula, F_spa and F_spe respectively represent the enhanced spatial feature and spectral feature, Concat represents the concatenation operation in the channel dimension, fconv represents the 5×5 convolutional operation, and F_ss represents the finally obtained concatenated feature.

[0125] The above feature fusion mechanism based on concatenation and convolution retains the complete information of the two features through feature concatenation, avoiding information loss that may be caused by direct addition; secondly, the use of a large-size convolutional kernel enhances the receptive field of feature fusion and helps to capture a wider range of feature correlations.

[0126] S303. Extract and fuse the multi-level features in the concatenated feature through the dense fusion module to obtain a target fusion feature.

[0127] After obtaining the stitching features, in order to further improve the expression ability of the features and make full use of the multi-scale information, it is necessary to perform deeper processing and optimization on the features. Since the targets in remote sensing images often have multi-scale characteristics, and the features at different scales contain complementary information, directly using the features of a single scale may not be able to comprehensively express the content of the image. Therefore, the embodiment of this application designs a dense fusion module including multiple sequentially arranged convolutional layers, and realizes multi-level extraction and deep fusion of features through dense connection.

[0128] Based on the above embodiment, as an optional embodiment, please refer to Figure 5 , Figure 5 FIG. is a schematic structural diagram of a dense fusion module provided by an embodiment of this application. The dense fusion module includes multiple convolutional layers. In S303: The step of extracting and fusing the multi-level features in the stitching features through the dense fusion module to obtain the target fusion features may specifically further include the following steps:

[0129] S601. Pass the stitching features through each of the convolutional layers in sequence to obtain multi-level features.

[0130] In the dense fusion process, in order to fully extract the multi-level information in the stitching features, it is necessary to process and optimize the features layer by layer through multiple convolutional layers. Since the remote sensing images contain feature information at different levels from local textures to global structures, the processing of a single convolutional layer may not be able to fully capture these multi-level features. Therefore, the embodiment of this application designs a series of sequentially arranged convolutional layers, and realizes the deep expression and effective transmission of features through layer-by-layer feature extraction and dense connection.

[0131] Specifically, the stitching features are sequentially input into each convolutional layer of the dense fusion module. Each convolutional layer not only receives the output features of the previous layer, but also obtains the feature information of all previous layers through dense connection. This design enables the network to extract and accumulate feature information at different levels, forming layer-by-layer enhancement and optimization of the features. Each convolutional layer uses the same convolutional kernel size, but can have different numbers of feature channels, so that while maintaining the same receptive field, the expression ability of the features can be gradually improved. In particular, the adjacent convolutional layers are connected in a dense manner, so that each layer can directly access the features of all previous layers. This feature reuse mechanism not only improves the utilization efficiency of the features, but also enhances the feature learning ability of the network.

[0132] The above feature extraction mechanism based on multi-layer convolution enables the network to gradually extract various feature patterns from low-level to high-level through the cascading of multiple convolutional layers; secondly, the design of dense connection ensures the efficient transmission and full utilization of feature information.

[0133] S602. Perform a splicing operation on the multi-level features to obtain the target fusion feature.

[0134] After obtaining the multi-level features, in order to effectively integrate the feature information of different levels, it is necessary to reasonably fuse the multi-level features. Since the features extracted by different convolutional layers in the dense fusion module have different spatial resolutions and semantic levels, directly performing feature fusion may lead to information misalignment and loss. For this reason, the embodiment of the present application designs a fusion strategy based on upsampling and feature splicing, and realizes the deep fusion of multi-level features by unifying the spatial resolution of the features and performing effective combination.

[0135] Specifically, first, perform upsampling on the features Fn output by each convolutional layer to make them have the same spatial resolution. The upsampling process uses the bicubic interpolation method, which can maintain the smoothness and continuity of the features while increasing the spatial resolution of the features, and avoids the blocky artifacts that may be brought by simple nearest neighbor interpolation. In particular, for deeper features, a larger upsampling factor is required to match the spatial size of the initial features. Subsequently, splice all the upsampled features in the channel dimension. This splicing operation retains the complete information of each layer of features. Finally, perform channel fusion and dimensionality reduction on the spliced features through a 3×3 convolutional layer to obtain the final target fusion feature.

[0136] Exemplarily, the feature fusion process can be expressed as:

[0137]

[0138] In the formula, UP(a, b) represents upsampling a with a scaling factor of b, and the bicubic interpolation method is used for the upsampling operation. Decoder() represents the decoding process, in which a 3×3 convolution is applied to restore the data to the image form.

[0139] The above feature fusion mechanism based on upsampling and splicing has significant advantages: First, the unified upsampling process ensures the alignment of each layer of features in the spatial dimension, facilitating subsequent feature fusion; Second, the splicing in the channel dimension retains the complete information of each layer of features, avoiding information loss.

[0140] S304. Decode the target fusion feature to obtain the target fusion image.

[0141] After obtaining the target fusion features, it is necessary to convert them into the final target fusion image. Since the target fusion features are high-dimensional representations in the feature space, containing rich spatial structures and spectral information, a decoding process is required to map them back to the image space to generate a fusion image with high spatial resolution and rich spectral information. For this purpose, the embodiments of the present application design a feature decoding strategy to achieve an effective conversion from the feature space to the image space through multi-level feature reconstruction and optimization.

[0142] Specifically, the decoding process first performs upsampling and feature transformation on the target fusion features through a series of transposed convolutional layers. Each transposed convolutional layer not only increases the spatial resolution of the features but also reorganizes the features through a learnable convolutional kernel, gradually converting the abstract feature representation into specific image information. In particular, a batch normalization layer and a ReLU activation function are added after each transposed convolutional layer. This design can not only stabilize the distribution of the features but also introduce appropriate non-linear transformation capabilities. To better preserve spatial details, a skip connection structure is also introduced in the decoding process to fuse the features of the corresponding layers in the encoding stage with the features of the current layer. This feature reuse mechanism helps to recover high-frequency detail information.

[0143] Based on the above embodiments, as an alternative embodiment, please refer to Figure 2 ... The panchromatic image, the multispectral image, and the prior image can be downsampled twice respectively to obtain three groups of different input images, and then the three groups of different input images are input into the remote sensing image spatial-spectral fusion network. Three target fusion features with different scales can be obtained in the remote sensing image spatial-spectral fusion network. Finally, the three target fusion features with different scales are stitched and decoded to obtain the final fusion features.

[0144] In the actual remote sensing image fusion process, image features at different scales often contain complementary information. Since ground object targets exhibit different feature patterns at different scales, feature extraction and fusion at a single scale may lose some important spatial and spectral details. To capture multi-scale feature information more comprehensively, the embodiments of the present application adopt a multi-scale feature extraction strategy, and the input image is processed by multiple downsamplings to improve the integrity and accuracy of the fusion effect.

[0145] Specifically, first, perform two downsampling operations on the panchromatic image, the multispectral image, and the prior image respectively to generate three groups of input images with different spatial resolutions. The downsampling process is implemented using average pooling because average pooling can maintain the main structural information of the image while reducing the image resolution, avoiding sudden loss of information or drastic changes in features. Through average pooling, each larger-scale feature map is converted into a smaller-scale feature map, forming a multi-scale feature pyramid structure. This multi-scale processing method enables the network to simultaneously focus on image features at different resolutions: the larger-scale feature maps mainly contain the global structural information of the image, while the smaller-scale feature maps retain more local detail information.

[0146] Subsequently, input these three groups of input images with different scales into the remote sensing image spatio-spectral fusion network respectively. At each scale level, the network will sequentially pass through the encoder, the multiple residual feature extraction module, the dual attention mechanism module, and the dense connection fusion module to perform feature extraction and fusion processing on the input images. This hierarchical processing method ensures that the network can effectively extract and integrate image features at each scale. In particular, the larger-scale features mainly focus on the overall structure and semantic information of the image, the medium-scale features balance local details and global structure, and the smaller-scale features are more focused on capturing fine texture and edge information.

[0147] After completing the extraction and fusion of features at each scale, the network will obtain three target fusion features with different scales. To integrate this multi-scale feature information, it is necessary to first resample the feature maps of different scales to the same spatial resolution. This application uses the bicubic interpolation method to implement the upsampling operation because, compared with nearest neighbor interpolation and bilinear interpolation, bicubic interpolation can produce a smoother interpolation result, reducing the aliasing effect and distortion phenomenon during the resampling process. Subsequently, the resampled features are concatenated in the channel dimension, and this concatenation operation retains the complete information of each scale feature. Finally, the decoder converts the concatenated multi-scale features into the final fused image, and the decoding process realizes the mapping conversion from features to the image domain through multi-layer convolution operations.

[0148] Based on the above embodiments, as an optional embodiment, please refer to Figure 6 , Figure 6 which is a schematic diagram of the construction of a loss function provided by the embodiment of this application. The target loss function includes:

[0149] The spatial loss function is used to calculate the structural similarity between the target fused image and the sample panchromatic image;

[0150] The spectral loss function is used to calculate the spectral angle mapping between the target fused image and the sample multispectral image;

[0151] A spatial spectral combination loss function is used to calculate the difference between a first residual probability distribution and a second residual probability distribution. The first residual probability distribution is the residual probability distribution between the target fused image and the sample panchromatic image, and the second residual probability distribution is the residual probability distribution between the sample multispectral image and the grayscale image corresponding to the sample multispectral image.

[0152] Among them, the spatial loss function can be derived from the structural similarity loss term, and the structural similarity loss term is obtained from the structural similarity index between the target fused image and the sample panchromatic image.

[0153] In the process of remote sensing image spatial-spectral fusion, maintaining the spatial structure consistency between the fused image and the panchromatic image is one of the key objectives. Since the human visual system is particularly sensitive to image structure information, directly using the mean square error at the pixel level may not accurately reflect the visual quality of the image. Therefore, the embodiment of this application introduces a structural similarity loss term, and by evaluating the structural similarity index (SSIM) between the target fused image and the sample panchromatic image, effectively constrains the reconstruction quality of the fused image in terms of spatial structure.

[0154] Specifically, the structural similarity index comprehensively evaluates the similarity degree of an image by considering the brightness, contrast, and structure of the image simultaneously. First, calculate the local statistical features between the target fused image IF and the sample panchromatic image IP, including the local mean, local standard deviation, and local covariance. These statistics are calculated within a local window to capture the structural information at an appropriate scale. Based on these statistics, the calculation of the SSIM index can be expressed as:

[0155]

[0156] In the formula, and respectively represent the mean values of images x and y, and respectively represent the standard deviations of images x and y, represents the covariance between images x and y, and are very small constants used to stabilize the output and avoid division-by-zero errors.

[0157] Among them, the spectral loss function can be derived from the spectral angle loss term, and the spectral angle loss term is obtained from the spectral angle mapping between the target fused image and the sample multispectral image.

[0158] In the process of remote sensing image spatial-spectral fusion, keeping the spectral characteristics of the fused image consistent with the original multi-spectral image is another key objective. Since the accuracy of spectral information directly affects subsequent remote sensing applications, such as land cover classification and change detection, etc., it is not enough to only focus on the reconstruction of the spatial structure. Therefore, the embodiment of this application introduces a spectral angle loss term, and by calculating the spectral angle mapping (SAM) between the target fused image and the sample multi-spectral image, effectively constrains the reconstruction quality of the fused image in the spectral dimension.

[0159] Specifically, the spectral angle mapping measures the similarity degree between two spectral vectors by calculating the included angle between them. This measurement method is particularly sensitive to changes in the spectral shape and is not affected by the spectral intensity. For the target fused image IF and the sample multi-spectral image IMS, their spectral angle can be expressed as:

[0160]

[0161] In the formula, where x⋅y represents the dot product between vectors x and y. ||·|| represents the Euclidean norm of the vector (i.e., the length of the vector).

[0162] Among them, the spatial-spectral combination loss function can be derived from the probability distribution loss term, and the probability distribution loss term is obtained from the KL divergence of the residual probability distribution between the target fused image and the sample panchromatic image, and the residual probability distribution between the sample spectral image and the corresponding grayscale image of the sample multi-spectral image.

[0163] In the process of remote sensing image spatial-spectral fusion, in addition to directly constraining the reconstruction quality of the spatial structure and spectral characteristics, it is also necessary to ensure that the statistical characteristics between the fused result and the source image are consistent. Since the probability distribution characteristics of the image reflect its inherent statistical laws, directly constraining the pixel values or feature values may not fully capture this statistical consistency. Therefore, the embodiment of this application introduces a probability distribution loss term, and by calculating the probability distribution difference of the residuals between the target fused image and the source image, constraints the rationality of the fusion process from a statistical perspective.

[0164] Exemplarily, its expression is as follows:

[0165]

[0166] Among them, p(x) represents the probability distribution function of the true information, and q(x) represents the probability distribution function of the fitted information. The KL divergence measures the distribution gap between p(x) and q(x) by comparing their differences. The smaller the value, the closer the distribution of the fitted information is to the true information.

[0167] In the process of remote sensing image spatial-spectral fusion, in order to optimize both spatial resolution and spectral fidelity simultaneously, a comprehensive loss function system needs to be designed. An embodiment of this application proposes a joint loss function, which includes a spatial loss function L_spa, a spectral loss function L_spe, and a spatial-spectral combined loss function L_css. Through these three complementary loss terms, the training process of the network is comprehensively guided.

[0168] Exemplarily, the spatial loss function L_spa optimizes the spatial resolution by evaluating the structural similarity between the target fused image and the panchromatic image PAN. Considering that the panchromatic image is single-channel while the target fused image is multi-channel, the PAN image first needs to be extended in the channel dimension to have the same number of channels as the target fused image. The calculation formula of the spatial loss function is:

[0169]

[0170] In the formula, the stack operation extends the single-channel PAN image to have the same number of channels as the target fused image. This loss term ensures that the fused result has spatial structural features similar to the panchromatic image in each band.

[0171] Exemplarily, the spectral loss function L_spe focuses on the spectral consistency between the target fused image and the original multi-spectral image MS. The spectral difference is measured by calculating the spectral angle mapping between the two images, and its calculation formula is:

[0172]

[0173] This loss term ensures that the fused result can maintain the spectral characteristics of the original multi-spectral image.

[0174] Furthermore, in order to establish the connection between spatial features and spectral features, a spatial-spectral combined loss function L_css is introduced. In order to construct a loss function that can comprehensively compare the target fused image with PAN and MS, the KL divergence is introduced into the loss function, based on the following assumptions:

[0175] a. A panchromatic image with the same resolution as the MS image can be obtained by converting the MS image into a grayscale image;

[0176] b. The grayscale image contains the spatial texture information of the source image;

[0177] c. The differences between the MS image and the PAN image at different spatial resolutions should follow a similar distribution.

[0178] Based on the above assumptions, first use the grayscale degradation network to generate a grayscale image from image . This network utilizes a channel-based attention mechanism through convolutional layers and global average pooling to highlight relevant channels and suppress irrelevant channels, compressing the multi-channel input into a single-channel result, thereby completing grayscale conversion. Subsequently, the output is stacked back into its original format, please refer to Figure 7 , Figure 7 which is a schematic diagram of the process of a grayscale degradation network provided by an embodiment of this application. The softmax function is used to construct a residual probability distribution map between the MS image and the grayscale image . This residual distribution is used to fit the residual probability distribution between the target fused image and the PAN. Finally, the KL divergence between these two probability distributions is minimized.

[0179] Exemplarily, the spatial-spectral combined loss function, and its specific formula is as follows:

[0180]

[0181]

[0182] In the formula, gray() represents the grayscale degradation network, is the grayscale image generated by through the degradation network.

[0183] The final combined loss function is obtained by weighted combination of these three loss terms:

[0184]

[0185] In the formula, the values of α, β, and γ are determined through experiments in the short-cycle training mode. By using SSIM, SAM, and KL divergence as loss functions, the network training process can be effectively optimized, thereby improving the fusion performance of the network.

[0186] The above training strategy based on multiple losses ensures the reconstruction quality of high-resolution details through the spatial loss; secondly, the spectral loss guarantees the accurate preservation of spectral information; finally, the spatial-spectral combined loss establishes the correlation between spatial and spectral features, providing a more comprehensive optimization target. Experimental results show that this multi-dimensional loss function design can effectively guide the training of the network, enabling the fused result to have both high spatial resolution and accurate spectral features simultaneously. Especially when dealing with complex scenes, it shows excellent balancing ability, and the generated fused image not only maintains clear spatial details but also accurately retains spectral information, providing high-quality data support for the practical application of remote sensing images.

[0187] Furthermore, in terms of the selection of training and test data, the embodiments of the present application adopt the WorldView-2 (WV-2) and Gaofen (GF) satellite datasets to evaluate the performance of the proposed fusion method. Specifically, the WV-2 dataset is used for the training and validation of the network, while the GF dataset series (including GF1, GF1B, GF1C, GF1D, GF2, GF6) is used to evaluate the generalization ability of the network. Since the WV-2 dataset contains more spectral bands than the GF-1 dataset, four bands (bands 2, 3, 4, and 7) in the WV-2 dataset are selected according to the spectral coverage range to ensure that its spectral range is consistent with that of the GF dataset. This selection enables the weights trained on the WV-2 dataset to be effectively transferred and applied to the GF dataset series, thereby improving the adaptability and generalization ability of the model on different satellite data.

[0188] Please refer to Figure 8 , Figure 8 FIG. is a schematic structural diagram of a self-supervised remote sensing image spatial-spectral fusion device based on prior image constraint provided by an embodiment of the present application. The self-supervised remote sensing image spatial-spectral fusion device based on prior image constraint includes:

[0189] An initial image acquisition module, configured to acquire a panchromatic image and a multispectral image;

[0190] A prior image generation module, configured to generate a prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, where the prior image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network for the panchromatic image and the multispectral image;

[0191] A target image fusion module, configured to input the panchromatic image, the multispectral image, and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fusion image output by the remote sensing image spatial-spectral fusion network;

[0192] Wherein, the remote sensing image spatial-spectral fusion network is trained in a self-supervised manner based on a target loss function, and the target loss function is constructed by calculating the probability distribution differences of the spatial loss and the spectral loss among the target fusion image, the sample panchromatic image, and the sample multispectral image.

[0193] Based on the above embodiment, as an optional embodiment, the prior image generation module is further configured to inject the spatial information in the panchromatic image into the multispectral image to obtain a prior image.

[0194] Based on the above embodiments, as an alternative embodiment, the target image fusion module is further configured to extract deep features from the panchromatic image, the multispectral image, and the prior image through the residual module; extract spatial features and spectral features from the deep features through the dual attention module, and extract and splice the spatial features and spectral features to obtain spliced features; extract and fuse multi-level features in the spliced features through the dense fusion module to obtain target fusion features; and decode the target fusion features to obtain a target fusion image.

[0195] Based on the above embodiments, as an alternative embodiment, the target image fusion module is further configured to encode the panchromatic image, the multispectral image, and the prior image respectively through an encoder to obtain encoded features; splice the encoded features corresponding to the panchromatic image, the multispectral image, and the prior image pairwise to obtain initial features; and process the initial features through the convolutional layer, the normalization layer, and the activation function layer to obtain deep features.

[0196] Based on the above embodiments, as an alternative embodiment, the target image fusion module is further configured to perform global max pooling operation and global average pooling operation on the deep features through the spatial attention sub-module, and obtain a spatial attention map through convolutional operation, and multiply the spatial attention map with the deep features to obtain spatial features; perform global average pooling operation on the deep features through the spectral attention sub-module, and obtain a spectral attention map through a convolutional layer, and multiply the spectral attention map with the deep features to obtain spectral features; and splice the spatial features and the spectral features to obtain spliced features.

[0197] Based on the above embodiments, as an alternative embodiment, the target image fusion module is further configured to sequentially pass the spliced features through each of the convolutional layers to obtain multi-level features; and perform a splicing operation on the multi-level features to obtain target fusion features.

[0198] Figure 9 The following is a schematic physical structure diagram of an electronic device provided by an embodiment of the present application. As Figure 9 shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940. Among them, the processor 910, the communication interface 920, and the memory 930 complete communication with each other through the communication bus 940. The processor 910 may call logical instructions in the memory 930 to execute a self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints.

[0199] In addition, when the logical instructions in the above-mentioned memory 930 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0200] On the other hand, this application also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints provided by the above-mentioned various methods.

[0201] On another aspect, this application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints provided by the above-mentioned various methods.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this application. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0203] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints, characterized in that, include: Acquire panchromatic and multispectral images; generating a priori image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, wherein the priori image is used to constrain the fusion process of the panchromatic image and the multispectral image by the remote sensing image spatial-spectral fusion network; Inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fused image output by the remote sensing image spatial-spectral fusion network; The remote sensing image spatial-spectral fusion network is trained in a self-supervised manner based on a target loss function, wherein the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image, and the sample multispectral image; The remote sensing image spatial-spectral fusion network includes a residual module, a dual attention module and a dense fusion module; The step of inputting the panchromatic image, the multispectral image, and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fused image output by the remote sensing image spatial-spectral fusion network includes: Performing feature extraction on the panchromatic image, the multispectral image, and the prior image by using the residual module to obtain depth features; Extracting spatial features and spectral features from the deep features through the dual attention module, and extracting and splicing the spatial features and spectral features to obtain spliced features; Extracting and fusing multi-level features in the splicing features through the dense fusion module to obtain target fusion features; The target fusion features are decoded to obtain a target fusion image.

2. The self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint according to claim 1, wherein The generating of a priori images based on the spatial information of the panchromatic image and the spectral information of the multispectral image comprises: The spatial information in the panchromatic image is injected into the multispectral image to obtain a priori image.

3. The self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint according to claim 1, wherein The residual module includes an encoder, a convolution layer, a normalization layer and an activation function layer; The extracting features of the panchromatic image, the multispectral image, and the prior image by the residual module to obtain depth features includes: Encoding the panchromatic image, the multispectral image, and the prior image respectively through an encoder to obtain encoding features; splicing the encoding features corresponding to the panchromatic image, the multispectral image, and the prior image in pairs to obtain initial features; The initial features are processed through the convolution layer, the normalization layer and the activation function layer to obtain deep features.

4. The self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint according to claim 1, wherein The dual attention module includes a spatial attention submodule and a spectral attention submodule; The extracting of spatial features and spectral features from the depth features by the dual attention module and splicing the spatial features and spectral features to obtain spliced features includes: Performing a global maximum pooling operation and a global average pooling operation on the depth feature through the spatial attention submodule, and obtaining a spatial attention map through a convolution operation, and multiplying the spatial attention map with the depth feature to obtain a spatial feature; Perform global average pooling operation on the depth features through the spectral attention sub-module, and obtain a spectral attention map through a convolutional layer. Multiply the spectral attention map with the depth features to obtain spectral features; Concatenate the spatial features and the spectral features to obtain concatenated features.

5. The self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints according to claim 1, characterized in that: The dense fusion module includes multiple convolutional layers; Extracting and fusing multi-level features in the concatenated features through the dense fusion module to obtain target fusion features, including: Pass the concatenated features through each of the convolutional layers in sequence to obtain multi-level features; Perform a concatenation operation on the multi-level features to obtain target fusion features.

6. The self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint according to claim 1, wherein The target loss function includes: A spatial loss function for calculating the structural similarity between the target fusion image and the sample panchromatic image; A spectral loss function for calculating the spectral angle mapping between the target fusion image and the sample multi-spectral image; A spatial-spectral combined loss function for calculating the difference between a first residual probability distribution and a second residual probability distribution. The first residual probability distribution is the residual probability distribution between the target fusion image and the sample panchromatic image, and the second residual probability distribution is the residual probability distribution between the sample multi-spectral image and the grayscale image corresponding to the sample multi-spectral image.

7. A self-supervised remote sensing image spatial-spectral fusion device based on prior image constraints, characterized in that: Including: An initial image acquisition module for acquiring a panchromatic image and a multi-spectral image; A prior image generation module for generating a prior image based on the spatial information of the panchromatic image and the spectral information of the multi-spectral image. The prior image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network for the panchromatic image and the multi-spectral image; A target image fusion module for inputting the panchromatic image, the multi-spectral image, and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fusion image output by the remote sensing image spatial-spectral fusion network; Wherein, the remote sensing image spatial-spectral fusion network is trained in a self-supervised manner based on a target loss function, and the target loss function is constructed by calculating the probability distribution differences of the spatial loss and the spectral loss between the target fusion image, the sample panchromatic image, and the sample multi-spectral image; The remote sensing image spatial-spectral fusion network includes a residual module, a dual attention module, and a dense fusion module; The target image fusion module, specifically for: Extract features of the panchromatic image, the multi-spectral image, and the prior image through the residual module to obtain depth features; Extract spatial features and spectral features from the depth features through the dual attention module, and extract and concatenate the spatial features and the spectral features to obtain concatenated features; Extract and fuse multi-level features in the concatenated features through the dense fusion module to obtain target fusion features; Decode the target fusion features to obtain a target fusion image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraint according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints as described in any one of claims 1 to 6 is implemented.

Images