Self-supervised remote sensing image space-spectrum fusion method and device based on prior image constraint

Abstract

The invention discloses a priori image constraint-based self-supervised remote sensing image space-spectrum fusion method and device, and relates to the technical field of remote sensing satellites. The method comprises the following steps: acquiring a panchromatic image and a multispectral image; a prior image is generated based on the spatial information of the panchromatic image and the spectral information of the multispectral image, and the prior image is used for constraining the fusion process of the panchromatic image and the multispectral image by a remote sensing image spatial-spectral fusion network; inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network, and obtaining a target fusion image output by the remote sensing image spatial-spectral fusion network; by implementing the technical scheme provided by the invention, the generalization ability of the model in processing different sensor data is enhanced, and a more stable fusion effect is realized.

Description

Technical Field

[0001] The present application relates to the field of remote sensing satellite technology, and in particular to a method and device for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints. Background Art

[0002] Remote sensing satellites can usually obtain panchromatic images and multispectral images at the same time. Panchromatic images have high spatial resolution but only contain single-band information, while multispectral images have rich spectral information but low spatial resolution. In order to obtain remote sensing images with both high spatial resolution and rich spectral information, it is necessary to perform spatial-spectral fusion processing on panchromatic images and multispectral images. Spatial-spectral fusion technology has important application value in agricultural monitoring, urban planning, environmental monitoring and other fields.

[0003] At present, deep learning-based spatial-spectral fusion methods mainly use supervised learning methods, which obtain training samples by constructing simulated downscaling data sets, use the original multispectral images as reference image labels, and perform supervised training on the fusion network. This type of method usually uses downscaled panchromatic images and multispectral images as network inputs, and optimizes network parameters by minimizing the error between the fusion results and the reference image. In addition, some studies have proposed the use of unsupervised learning methods such as generative adversarial networks to directly use images with original resolution for training.

[0004] However, existing deep learning methods often lack sufficient consideration of the inherent relationship between panchromatic images and multispectral images when performing spatial-spectral fusion. Due to the differences in spectral response and spatial resolution between images collected by different sensors, it is difficult to accurately capture this relationship by relying solely on the network's own feature extraction capabilities, which can easily lead to distortion in the preservation of spatial details and spectral information in the fusion results. In particular, when processing data from different sensors, the model's generalization ability is poor and it is impossible to maintain a stable fusion effect. Summary of the invention

[0005] The present application provides a method and device for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints, which enhances the generalization ability of the model when processing data from different sensors and achieves a more stable fusion effect.

[0006] In a first aspect of the present application, a self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints is provided, comprising: Acquire panchromatic and multispectral images; Based on the spatial information of the panchromatic image and the spectral information of the multispectral image, a priori image is generated, wherein the priori image is used to constrain the fusion process of the panchromatic image and the multispectral image by the remote sensing image spatial-spectral fusion network; Inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain the remote sensing image spatial-spectral fusion network output target fused image; Among them, the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image and the sample multispectral image.

[0007] In a second aspect of the present application, a self-supervised remote sensing image spatial-spectral fusion device based on prior image constraints is provided, comprising: An initial image acquisition module, used to acquire full-color images and multispectral images; A priori image generation module, used to generate a priori image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, wherein the priori image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network on the panchromatic image and the multispectral image; A target image fusion module is used to input the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fused image output by the remote sensing image spatial-spectral fusion network; Among them, the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image and the sample multispectral image.

[0008] In a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints as described above is implemented.

[0009] In a fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints as described in any one of the above is implemented.

[0010] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By introducing a prior image as a constraint, the prior image contains the spatial information of the panchromatic image and the spectral information of the multispectral image, providing the remote sensing image space-spectral fusion network with prior knowledge of the inherent relationship between images. At the same time, the prior image is input into the fusion network together with the panchromatic image and the multispectral image, and a self-supervised training method is adopted. The target loss function is used to calculate the probability distribution difference of the spatial loss and spectral loss between the target fusion image, the sample panchromatic image, and the sample multispectral image, so that the network can fully learn and utilize the inherent relationship between images during the training process. This self-supervised learning method based on prior constraints avoids the limitation of relying solely on the feature extraction ability of the network itself, improves the accuracy of the fusion result in maintaining spatial details and spectral information, and enhances the generalization ability of the model when processing different sensor data, achieving a more stable fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 It is a flowchart of a method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints provided in an embodiment of the present application; Figure 2 This is a schematic diagram of a remote sensing image spatial-spectral fusion network framework provided in an embodiment of the present application; Figure 3 is a schematic diagram of the structure of a residual module provided in an embodiment of the present application; Figure 4 is a structural schematic diagram of a dual attention module provided in an embodiment of the present application; Figure 5 is a structural diagram of a dense fusion module provided in an embodiment of the present application; Figure 6 It is a schematic diagram of the construction of a loss function provided in an embodiment of the present application; Figure 7 It is a process schematic diagram of a grayscale degradation network provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a self-supervised remote sensing image spatial-spectral fusion device based on a priori image constraints provided in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0014] Please refer to Figure 1 , Figure 1 This is a flowchart of a method for self-supervised remote sensing image spatial-spectral fusion based on a priori image constraints provided by an embodiment of the present application. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, and can also be run on a self-supervised remote sensing image spatial-spectral fusion system based on a priori image constraints based on a von Neumann system. The computer program can be integrated into an application or run as an independent tool application. Specifically, the method may include the following steps: S101. Acquire a panchromatic image and a multispectral image.

[0015] Among them, panchromatic images refer to single-band images acquired by sensors in a wider band range. Panchromatic images are used to provide spatial structural information of ground objects, including high-frequency detail information such as texture features and edge contours.

[0016] Correspondingly, multispectral images refer to multi-band images obtained simultaneously by sensors in multiple independent narrow bands. Multispectral images are used to provide spectral reflectance characteristics of objects, which helps to distinguish the physical and chemical properties of different objects and realize object classification and identification.

[0017] Specifically, before performing spatial-spectral fusion of remote sensing images, it is necessary to first obtain the panchromatic image and multispectral image to be fused. Due to the physical limitations of satellite sensors, optical remote sensing images acquired by the same satellite usually only have advantages in one aspect, namely spatial resolution or spectral resolution: panchromatic images have high spatial resolution but limited spectral information, and multispectral images have rich spectral information but low spatial resolution. Therefore, it is necessary to fuse these two complementary image information to obtain a fused image with both high spatial resolution and rich spectral information.

[0018] S102, generating a priori image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, wherein the priori image is used to constrain the fusion process of the panchromatic image and the multispectral image by the remote sensing image spatial-spectral fusion network.

[0019] Among them, the prior image refers to a reference image generated by reasonably combining the spatial information of the panchromatic image and the spectral information of the multispectral image. The prior image is used to provide training reference and constraint guidance for the remote sensing image spatial-spectral fusion network, helping the network to better balance the preservation of spatial detail information and spectral feature information during the self-supervised learning process, thereby improving the quality of the fusion result.

[0020] Specifically, before fusing the panchromatic image and the multispectral image, a prior image needs to be generated first. Due to the lack of real high-resolution multispectral images as training references, traditional supervised learning methods often require manual labeling or simulated degraded data sets, which not only increases the workload but also may cause the loss of spatial information. Therefore, this application proposes to use prior images to constrain the fusion process, and to generate a prior image with reference value by reasonably combining the spatial information of the panchromatic image and the spectral information of the multispectral image.

[0021] Based on the above embodiment, as an optional embodiment, the spatial information in the full-color image can be injected into the multi-spectral image to obtain a priori image.

[0022] Specifically, although the panchromatic image has only a single band, its spectral response range is wide and includes the comprehensive reflectance characteristics of the objects in multiple bands. Therefore, the spatial information contained in the panchromatic image can be transferred to the multispectral image by establishing a statistical relationship between the panchromatic image and the bands of the multispectral image.

[0023] In a feasible implementation, the Gram-Schmidt (GS) method can be used, which first performs mathematical orthogonalization on the multispectral data, replaces the first orthogonal basis with the full-color image, and then performs an inverse transformation to generate a priori image. The above method can make full use of the spatial information in the full-color image while maintaining the original spectral characteristics of the multispectral image, thereby improving the spectral fidelity of the fusion result.

[0024] In another feasible implementation, the nearest neighbor diffusion (NND) method can be used, which first separates the high-frequency and low-frequency information of the panchromatic image through low-pass filtering, then calculates the similarity weights between the panchromatic image and each band of the multispectral image, and finally adaptively injects the high-frequency information of the panchromatic image into the multispectral image based on these weights. The above method can not only effectively maintain the spectral features in the multispectral image, but also adaptively adjust the degree of information injection according to the correlation between different bands and the panchromatic image, avoiding the problem of over-enhancement or under-enhancement.

[0025] The Gram-Schmidt (GS) method is used to generate a priori images. The GS method aligns the spectral characteristics of the full-color image and the multispectral image through an orthogonalization process, which can minimize the spectral distortion while improving the spatial resolution, thereby achieving a better fusion effect.

[0026] Specifically, the multispectral image is first upsampled to make its spatial resolution consistent with that of the panchromatic image. Then, the upsampled multispectral image and the panchromatic image are spliced ​​in the spectral dimension to form an image sequence to be processed. Next, the GS orthogonalization transformation is performed on the image sequence to obtain a set of orthogonal bases. In the orthogonal basis, the first orthogonal basis is replaced with the panchromatic image, and the other orthogonal bases are kept unchanged. Finally, the GS inverse transformation is performed on the replaced orthogonal basis to obtain the prior image.

[0027] For example, the Gram-Schmidt spatial-spectral fusion method can be concisely expressed as the following formula: In the formula, cat(a,b) means concatenating a and b along the spectral band; Indicates replacement The GS method mathematically orthogonalizes the MS data and performs an inverse transform. Specifically, the method first orthogonalizes the MS data, replaces the first orthogonal basis with the PAN image, and then performs an inverse transform to generate a priori images.

[0028] The above-mentioned spatial information injection method based on GS transformation can effectively maintain the spectral characteristics of multispectral images, and at the same time reasonably distribute the spatial detail information of panchromatic images to each spectral band. Through orthogonalization processing, the GS method reduces the correlation between bands and reduces spectral distortion, so that the generated prior image has good spectral fidelity. This provides a reliable basis for subsequent spatial information injection and helps to improve the quality of fusion results.

[0029] The nearest neighbor diffusion (NND) method is used to generate a priori images. The NND method uses the nearest neighbor search and local similarity principles to effectively transfer the high-frequency information in the panchromatic image to the multispectral image while keeping the spectral features without obvious distortion.

[0030] Specifically, the multispectral image is first upsampled to make its spatial resolution consistent with that of the panchromatic image. Then, high-frequency detail information is extracted from the panchromatic image, including texture features and edge contours. Next, for each band of the upsampled multispectral image, the image block that is most similar to the current processing area is searched in the panchromatic image to establish a corresponding relationship. Based on this corresponding relationship, the high-frequency information in the panchromatic image is gradually injected into each band of the multispectral image by local mean diffusion, and finally a priori image is obtained.

[0031] For example, the NND algorithm uses pixel-level similarity and diffusion process to effectively propagate spatial details in the image. The algorithm extracts high-frequency information from the P image and the MS image respectively, and its implementation process is as follows: Wherein, LPF(·) represents a low-pass filter operation function. express The i-th band of . , and Represents the low-frequency information of the P image, the high-frequency information of the P image, and Therefore, the weights of P and MS at each pixel position (x, y) and band i are It can be calculated as follows: Where Z(x,y) is a normalization constant such that .parameter Controls how much weights are decayed.

[0032] Then, by weight Inject the high-frequency details of the P image into the multispectral bands, and finally get the fused output: In the formula, C represents the number of bands of MS, Represents the final fusion result. The NND method uses the spatial information of the PAN image to guide the enhancement of the MS data. Specifically, it calculates the similarity between the PAN image and each MS band in the low-frequency domain, assigns adaptive weights based on the similarity, and then injects the high-frequency details of the PAN image into the MS data. The resulting prior image retains both the spatial details of the PAN image and the spectral fidelity of the MS band.

[0033] Based on the above embodiments, after generating the prior image by the GS and NND methods, considering that the ground objects in the remote sensing image often have multi-scale characteristics, extracting features only at a single scale may lose some important spatial and spectral information. Therefore, multi-scale feature extraction can be used to fully capture the image details at different scales, thereby improving the integrity and accuracy of the fusion effect.

[0034] Specifically, the multi-scale feature extraction process first performs multiple downsampling operations on the full-color image, multi-spectral image, and prior image. This application uses average pooling to achieve downsampling, because average pooling can maintain the main structural information of the image while reducing the image resolution, avoiding sudden loss of information or drastic changes in features. Through average pooling, each larger-scale feature map is converted into a smaller-scale feature map, forming a multi-scale feature pyramid structure.

[0035] At each scale level, the network performs feature extraction and fusion operations on the downsampled full-color image, multispectral image, and prior image. This hierarchical processing method enables the network to focus on image features at different scales at the same time: the feature map of the larger scale mainly contains the global structural information of the image, while the feature map of the smaller scale retains more local detail information. By processing in parallel at different scales, the network can learn a richer and more complete image feature representation.

[0036] After the extraction and fusion of features at each scale, the feature maps of different scales need to be resampled to the same spatial resolution. This application uses the bicubic interpolation method to implement the upsampling operation. This is because compared with nearest neighbor interpolation and bilinear interpolation, bicubic interpolation can produce smoother interpolation results and reduce the jagged effect and distortion during the resampling process. Through bicubic interpolation, the feature maps of smaller scales are resampled to larger scales, maintaining the continuity and smoothness of the features.

[0037] S103, inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network, obtaining the remote sensing image spatial-spectral fusion network output target fusion image, wherein the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss among the target fusion image, the sample panchromatic image and the sample multispectral image.

[0038] Among them, the remote sensing image spatial-spectral fusion network refers to a neural network structure based on deep learning, which can simultaneously process three inputs of panchromatic images, multispectral images and prior images, and realize the fusion of high spatial resolution and high spectral resolution images through the synergy of multiple functional modules.

[0039] In the embodiment of the present application, it can be understood as a three-stream fusion network architecture consisting of a residual module, a dual attention module and a dense fusion module. Among them, the residual module is responsible for feature extraction and mapping, and enhances the feature learning ability of the network through the residual connection structure; the dual attention module contains two sub-modules, spatial attention and spectral attention, which focus on the reconstruction of spatial details and spectral features respectively; the dense fusion module realizes the effective fusion of multi-scale features through dense connections.

[0040] Furthermore, the remote sensing image spatial-spectral fusion network is used to fuse multispectral images with low spatial resolution but rich spectral information with panchromatic images with high spatial resolution but limited spectral information, and at the same time use the prior image to provide additional constraint information, and finally generate a target fused image that maintains both high spatial resolution and rich spectral information. The network is trained through self-supervised learning, without the need for manual training labels, and can better maintain the characteristic information of the original data and improve the quality of the fusion results.

[0041] Specifically, after obtaining the prior image, the panchromatic image, multispectral image and prior image need to be input into the remote sensing image spatial-spectral fusion network to obtain the target fused image. Due to the lack of real high-resolution multispectral images as training labels, traditional supervised learning methods often need to train the network by manually making labels or simulating degraded data sets, which not only increases the workload but also may introduce additional errors. In order to solve this problem, the embodiment of the present application adopts a self-supervised learning method to train the remote sensing image spatial-spectral fusion network, directly using the input panchromatic image, multispectral image and prior image as training references.

[0042] For example, please refer to Figure 2 , Figure 2 A schematic diagram of a remote sensing image spatial-spectral fusion network framework provided in an embodiment of the present application is shown.

[0043] Specifically, in the remote sensing image spatial-spectral fusion network architecture, the MS image is first upsampled to generate an upsampled multispectral image (UPMS). In order to better adapt to different sensor data, the traditional spatial-spectral fusion method is used to fuse the UPMS with the PAN image to obtain a prior image. The prior image, together with PAN and MS, enters the network as a three-stream input. In order to more effectively extract deep features, the three-stream input data is downsampled multiple times through the average pooling layer (AP) to generate a multi-scale input image. At each scale, the data passes through the encoder, the multi-residual feature extraction module (MRFEM), the dual attention mechanism module (DAM), and the densely connected fusion module (DenseFuse) in turn to obtain the target fusion features at that scale. Finally, these target fusion features are resampled to the same scale through bicubic interpolation, spliced, and processed by the decoder to generate a high-resolution multispectral image. Finally, a self-supervised learning method is used to guide the training of the network by combining spatial, spectral and spatial-spectral joint loss functions to ensure better feature fusion from both spatial and spectral dimensions.

[0044] Exemplarily, the SPTN model can be expressed as: Where, f(.) represents the SPTN network, Represents numerical values ​​related to different parameter settings in the network. , and as the input of the network.

[0045] The remote sensing image spatial-spectral fusion network uses a number of advanced architectural features to ensure the best feature extraction and fusion quality. The key features include: a multi-scale module, which is used to capture features at different resolutions through sensors to ensure that the model can identify details at all scales; a residual module, which enhances the depth and efficiency of learning and enables the model to extract more complex features; a dual attention module, which selectively focuses on spatial and spectral features, thereby more effectively balancing the retention of spatial resolution and spectral information; and a dense fusion module, which achieves full fusion of features through dense connections within the network, ensuring rich integration of features at different levels and improving the accuracy and integrity of the fusion effect.

[0046] The above network training method based on self-supervised learning avoids the tedious process of manually making training labels, directly uses the input image as a supervision signal, and better maintains the characteristics of the original data. At the same time, the multi-module cascade network architecture and multi-dimensional loss function design enable the network to fully learn the balance between spatial details and spectral features, effectively improving the quality of the fused image. In particular, the introduction of the dual attention mechanism and dense connection structure enhances the network's perception of important features and improves the effect of feature extraction and fusion.

[0047] Based on the above embodiment, as an optional embodiment, the remote sensing image spatial-spectral fusion network includes a residual module, a dual attention module and a dense fusion module; in S103: the panchromatic image, the multispectral image and the prior image are input into the remote sensing image spatial-spectral fusion network, and the step of obtaining the remote sensing image spatial-spectral fusion network output target fusion image may further include the following steps: S301, extracting features from the full-color image, the multispectral image and the prior image through a residual module to obtain depth features.

[0048] Specifically, in order to effectively extract and fuse the feature information of the input image, it is first necessary to perform feature extraction on the panchromatic image, multispectral image and prior image through a residual module. Since the three input images have different feature distributions and data structures, direct feature fusion may result in information loss or distortion. In order to solve this problem, the embodiment of the present application designs a residual module including an encoder, a convolutional layer, a normalization layer and an activation function layer, and realizes the extraction of different image features through feature mapping and residual learning.

[0049] Based on the above embodiment, as an optional embodiment, please refer to Figure 3 , Figure 3 A schematic diagram of the structure of a residual module provided in an embodiment of the present application, wherein the residual module includes an encoder, a convolution layer, a normalization layer, and an activation function layer; in S301: the step of mapping the spectral features in the full-color image, the multispectral image, and the prior image to the spatial features through the residual module to obtain the depth features may further include the following steps: S401, respectively encode the full-color image, the multispectral image and the prior image through an encoder to obtain encoding features.

[0050] In the process of remote sensing image spatial-spectral fusion, since panchromatic images, multispectral images, and prior images have different data structures and feature distributions, direct feature extraction and fusion may result in information loss or feature misalignment. In order to solve this problem, it is necessary to first map the three input images to a unified feature space. The embodiment of the present application performs feature encoding on different input images by designing a special encoder to ensure that subsequent feature extraction and fusion can be performed in the same feature domain.

[0051] Specifically, the encoder uses a multi-layer convolutional network structure to extract and transform features based on the characteristics of different input images. For full-color images, the encoder focuses on extracting spatial structural information, including high-frequency details such as texture features and edge contours; for multispectral images, the encoder focuses on maintaining its rich spectral information while improving its spatial expression capabilities; for prior images, the encoder needs to pay attention to information in both spatial and spectral dimensions to provide effective fusion constraints. Through this targeted design, it is ensured that the encoding process can fully retain the important features of various input images.

[0052] Exemplarily, the encoding process can be expressed as: E = Encoder (Input); In the formula, E represents the encoded feature, Input is the input data (such as a panchromatic image, a multispectral image, or a priori image), and Encoder(·) represents the encoder function. By applying the encoding process, the encoded representation EP of the panchromatic image, the encoded representation EMS of the multispectral image, and the encoded representation EPI of the priori image are obtained respectively. These encoded features have the same dimension and feature expression, providing a unified basis for subsequent feature fusion.

[0053] The above encoder-based feature conversion method can effectively extract the key features of different types of images through a specially designed encoding structure; secondly, it maps the features of different domains into a unified feature space, solving the basic problem of feature fusion.

[0054] S402, concatenating the encoding features corresponding to the panchromatic image, the multispectral image, and the priori image in pairs to obtain initial features.

[0055] After obtaining the coding features of the full-color image, multispectral image, and prior image, these features need to be spliced ​​to make full use of the complementary information between different images. Since the three input images contain different key information: the full-color image provides high-resolution spatial details, the multispectral image provides rich spectral information, and the prior image provides fusion references and constraints, directly splicing the three features at one time may cause mutual interference of information or weakening of features. In order to better maintain the advantages of various features and achieve information complementarity, the embodiment of the present application adopts a paired splicing strategy to gradually extract and splice effective information through the combination of two features.

[0056] Specifically, the encoded features EP, EMS and EPI are combined in pairs to splice the encoded features of the panchromatic image and the multispectral image, the encoded features of the multispectral image and the prior image, and the encoded features of the panchromatic image and the prior image. The splicing process of each pair of features first performs splicing in the channel dimension, and then realizes the interaction between features and information extraction through convolution operations. This pair-wise splicing method can more finely control the information flow between different types of features, ensuring that important features will not be excessively weakened or lost during the splicing process.

[0057] Exemplarily, the pairwise feature concatenation process can be expressed as: Among them, cat_each(a,b,c) represents the channel splicing of a, b and c in pairs, and then the overall splicing. F represents the initial features obtained after splicing, and MRFEM() represents the operation of feature extraction and splicing using multiple deep residual modules. This design ensures that each pair of features can fully exchange information and reorganize features.

[0058] The above-mentioned feature processing based on paired splicing reduces the information interference that may occur when directly fusing three features by combining two features. Secondly, the splicing of each pair of features can focus on a specific information complementary relationship. For example, the splicing of panchromatic images and multispectral images focuses on the integration of spatial-spectral information, and the splicing of multispectral images and prior images pays more attention to the constraint and optimization of features. Finally, the splicing results of multiple pairs of features together constitute a more complete feature representation, providing a rich information basis for subsequent deep feature extraction. Experimental results show that this paired splicing strategy can better maintain the dominant features of various types of images, while achieving effective complementarity between different features.

[0059] S403, processing the initial features through a convolution layer, a normalization layer, and an activation function layer to obtain a deep feature.

[0060] After obtaining the initial features, in order to further extract and optimize the feature representation, it is necessary to perform deep processing and conversion of the features through a deep network structure. Since the initial features mainly contain the basic spatial structure and spectral information, it is still necessary to capture more complex feature patterns and hierarchical relationships through deeper feature extraction. To this end, the embodiment of the present application designs a processing unit composed of a convolutional layer, a normalization layer, and an activation function layer, and through the synergy of these basic components, deep extraction and optimization of features are achieved.

[0061] Specifically, the initial features are first extracted in the spatial domain through the convolution layer. The convolution kernel captures the feature patterns of the local area through a sliding window, including spatial structural information such as texture and edge. Next, the features are standardized using the normalization layer. This step can reduce the internal offset of the feature distribution and improve the training stability and convergence speed of the network. Finally, the nonlinear transformation capability is introduced through the activation function layer, so that the network can learn more complex feature mapping relationships. In particular, the residual connection structure is introduced in these processing units to directly add the input features to the output features. This design can not only alleviate the gradient vanishing problem in deep network training, but also maintain the important information in the original features.

[0062] Exemplarily, the processing process of the depth residual block can be expressed as: In the formula, F n represents the output feature of the nth layer, ω n is the weight parameter of this layer, F n-1 is the output feature of the previous layer, and bn is the bias parameter. By connecting multiple such residual blocks in series, the network can learn more abstract and expressive feature representations layer by layer, and finally obtain deep features containing rich semantic information. Each residual block not only learns the residual part of the feature, but also retains important information in the original feature through jump connections, ensuring the effective transmission of information in the deep network.

[0063] The above-mentioned feature extraction method based on multi-layer processing units, including the multi-layer stacking of convolutional layers, enables the network to gradually extract various feature patterns from low layers to high layers; secondly, the introduction of normalization layers improves the stability and efficiency of network training; thirdly, the nonlinear activation function enhances the feature expression ability of the network.

[0064] S302. Extracting spatial features and spectral features from the deep features through the dual attention module, and splicing the spatial features and spectral features to obtain spliced ​​features.

[0065] After obtaining the deep features, in order to more effectively utilize the spatial information and spectral information in the features, a mechanism that can adaptively focus on different feature dimensions is needed. Since the core goal of spatial-spectral fusion of remote sensing images is to simultaneously maintain high spatial resolution and rich spectral information, directly processing deep features may ignore certain important feature expressions. To this end, the embodiment of the present application designs a dual attention module including a spatial attention submodule and a spectral attention submodule. Through two independent but collaborative attention mechanisms, the expression of spatial features and spectral features are enhanced respectively, thereby achieving effective feature extraction and splicing.

[0066] Based on the above embodiment, as an optional embodiment, please refer to Figure 4 , Figure 4 A schematic diagram of the structure of a dual attention module provided in an embodiment of the present application, wherein the dual attention module includes a spatial attention submodule and a spectral attention submodule; in S302: extracting spatial features and spectral features from the deep features by the dual attention module, and splicing the spatial features and spectral features to obtain spliced ​​features, this step may further include the following steps: S501. Perform global maximum pooling and global average pooling operations on the deep features through the spatial attention submodule, obtain a spatial attention map through convolution operation, and multiply the spatial attention map with the deep features to obtain spatial features.

[0067] In the process of spatial-spectral fusion of remote sensing images, effective extraction and enhancement of spatial features are crucial to maintaining the detailed information and structural features of the image. Since deep features contain rich but unevenly distributed spatial information, direct processing may ignore some key spatial structures. To this end, the embodiment of the present application designs a spatial attention submodule, which adaptively generates a spatial attention map by simultaneously considering the maximum response and average distribution of the feature map, thereby highlighting important spatial area features.

[0068] Specifically, the spatial attention submodule first performs global maximum pooling and global average pooling operations on the input deep features. Global maximum pooling can capture the most significant activation response in the feature map, which helps to locate important spatial structures and edge information; global average pooling provides overall statistical information of the feature map, reflecting the general feature distribution of different regions. These two complementary pooling operations jointly construct a more comprehensive description of spatial features. Subsequently, the results of the two pooling operations are concatenated in the channel dimension, and the features are integrated through a 3×3 convolutional layer. Finally, the Sigmoid activation function is used to map the features to between 0 and 1 to generate a spatial attention map. The value of each position in the spatial attention map represents the importance of the position in the spatial dimension. Finally, the spatial attention map is element-wise multiplied with the original deep features to achieve feature enhancement of important spatial regions.

[0069] Exemplarily, the calculation process of the spatial attention mechanism can be expressed as: Where GMP() and GAP() represent the global maximum pooling and global average pooling operations respectively. represents the Sigmoid activation function, Represents an element-by-element multiplication operation. Through the above calculation method, the spatial attention submodule can generate an attention map S that reflects the importance of space, and use it to modulate the original feature F to obtain the enhanced spatial feature F_spa.

[0070] The above-mentioned feature enhancement mechanism based on spatial attention provides a more comprehensive description of spatial features by combining the two operations of maximum pooling and average pooling. Secondly, the adaptively generated attention map can dynamically adjust the focus area according to the specific circumstances of the input features. Finally, the soft weighting method of the attention mechanism avoids the information loss that may be caused by hard threshold division.

[0071] S502, performing a global average pooling operation on the deep features through the spectral attention submodule, and obtaining a spectral attention map through a convolutional layer, and multiplying the spectral attention map with the deep features to obtain spectral features.

[0072] In the process of spatial-spectral fusion of remote sensing images, maintaining and enhancing spectral features is of great significance to ensure the spectral fidelity of the fusion results. Since the spectral information in the deep features is distributed in different feature channels and the importance of each channel varies, direct processing may weaken key spectral features. To this end, the embodiment of the present application designs a spectral attention submodule, which adaptively generates a spectral attention map by establishing dependencies between feature channels, thereby highlighting important spectral features.

[0073] Specifically, the spectral attention submodule first performs a global average pooling operation on the input deep features. Global average pooling can extract global statistical information for each feature channel and effectively capture the overall response characteristics of different spectral channels. This pooling operation compresses the two-dimensional spatial features into a one-dimensional channel descriptor, allowing the network to better focus on the feature distribution at the channel level. Subsequently, the pooled features are transformed through two 1×1 convolutional layers in sequence. The first convolutional layer is responsible for dimensionality reduction to reduce the amount of calculation and extract the correlation between channels, and the second convolutional layer restores the features to the original number of channels. This dimensionality reduction and dimensionality increase operation effectively models the nonlinear relationship between channels. Finally, the Sigmoid activation function is used to map the features to between 0 and 1 to generate a spectral attention map. Each value in the spectral attention map represents the importance of the corresponding channel in the spectral dimension. Finally, the spectral attention map is element-wise multiplied with the original deep features to enhance the important spectral features.

[0074] Exemplarily, the calculation process of the spectral attention mechanism can be expressed as: In the formula, F represents the deep features of the input, GAP() represents the global average pooling operation, fconv represents the series operation of two 1×1 convolutional layers, σ represents the Sigmoid activation function, Represents an element-by-element multiplication operation. Through this calculation method, the spectral attention submodule can generate an attention map Q that reflects the importance of the channel, and use it to modulate the original feature F to obtain the enhanced spectral feature F_spe.

[0075] The above-mentioned feature enhancement mechanism based on spectral attention has significant advantages: first, global information at the channel level is obtained through global average pooling, which effectively characterizes the feature distribution of different spectral channels; second, the design of two layers of convolution fully models the dependency between channels through dimensionality reduction and dimensionality increase operations; finally, the adaptively generated attention weights can dynamically adjust the importance of different channels according to the input features.

[0076] S503: Concatenate the spatial features and the spectral features to obtain concatenated features.

[0077] After obtaining the enhanced spatial features and spectral features respectively, it is necessary to effectively splice these two complementary features to obtain a feature representation with both high-quality spatial expression and rich spectral information. Since the spatial feature F_spa mainly contains the structural detail information of the image, and the spectral feature F_spe retains the rich spectral response characteristics, simply superimposing or splicing the two features may cause mutual interference or information redundancy between the features. To this end, the embodiment of the present application designs a feature splicing strategy, which realizes the effective integration of spatial features and spectral features through a combination of feature splicing and convolution transformation.

[0078] Specifically, the spatial features F_spa and spectral features F_spe, which have been enhanced by the spatial attention submodule and the spectral attention submodule respectively, are first concatenated in the channel dimension. This concatenation operation retains the complete information of the two features, so that subsequent processing can make full use of the complementary relationship between the two features. Subsequently, the concatenated features are processed through a 5×5 convolution layer. The larger convolution kernel size helps to capture a wider range of feature dependencies and achieve deep fusion between features. This convolution operation can not only learn the nonlinear mapping relationship between spatial features and spectral features, but also reduce the feature dimension, remove redundant information, and generate a more compact and effective feature representation.

[0079] Exemplarily, the feature fusion process can be expressed as: In the formula, F_spa and F_spe represent the enhanced spatial features and spectral features respectively, Concat represents the concatenation operation in the channel dimension, fconv represents the 5×5 convolution operation, and F_ss represents the final concatenation feature.

[0080] The above-mentioned feature fusion mechanism based on splicing and convolution maintains the complete information of the two features through feature splicing, avoiding the information loss that may be caused by direct addition; secondly, the use of large-size convolution kernels enhances the receptive field of view of feature fusion and helps to capture a wider range of feature associations.

[0081] S303, extracting and fusing multi-level features in the splicing features through the dense fusion module to obtain target fusion features.

[0082] After obtaining the spliced ​​features, in order to further improve the expressive power of the features and make full use of multi-scale information, the features need to be processed and optimized at a deeper level. Since the targets in remote sensing images often have multi-scale characteristics, and features of different scales contain complementary information, directly using features of a single scale may not be able to fully express the content of the image. To this end, the embodiment of the present application designs a dense fusion module comprising multiple sequentially arranged convolutional layers, which realizes multi-level extraction and deep fusion of features through dense connections.

[0083] Based on the above embodiment, as an optional embodiment, please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a dense fusion module provided in an embodiment of the present application, wherein the dense fusion module includes multiple convolutional layers; in S303: extracting and fusing multi-level features in the splicing features through the dense fusion module to obtain the target fusion feature, the step may further include the following steps: S601, passing the concatenated features through each of the convolutional layers in sequence to obtain multi-level features.

[0084] In the dense fusion process, in order to fully extract the multi-level information in the spliced ​​features, it is necessary to process and optimize the features layer by layer through multiple convolutional layers. Since remote sensing images contain feature information at different levels, from local texture to global structure, the processing of a single convolutional layer may not be able to fully capture these multi-level features. To this end, the embodiment of the present application designs a series of sequentially arranged convolutional layers to achieve deep expression and effective transmission of features through layer-by-layer feature extraction and dense connection.

[0085] Specifically, the concatenated features are input into each convolutional layer of the dense fusion module in sequence. Each convolutional layer not only receives the output features of the previous layer, but also obtains the feature information of all previous layers through dense connections. This design enables the network to extract and accumulate feature information at different levels, forming a layer-by-layer enhancement and optimization of features. Each convolutional layer uses the same convolution kernel size, but can have a different number of feature channels, so that the expressiveness of the features can be gradually improved while maintaining the consistency of the receptive field. In particular, dense connections are used between adjacent convolutional layers, so that each layer can directly access the features of all previous layers. This feature reuse mechanism not only improves the efficiency of feature utilization, but also enhances the feature learning ability of the network.

[0086] The above-mentioned feature extraction mechanism based on multi-layer convolution enables the network to gradually extract various feature patterns from low to high layers through the series connection of multiple convolutional layers; secondly, the dense connection design ensures the efficient transmission and full utilization of feature information.

[0087] S602: performing a splicing operation on the multi-level features to obtain a target fusion feature.

[0088] After obtaining multi-level features, in order to effectively integrate feature information at different levels, it is necessary to reasonably fuse the multi-level features. Since the features extracted by different convolutional layers in the dense fusion module have different spatial resolutions and semantic levels, direct feature fusion may cause misalignment and loss of information. To this end, the embodiment of the present application designs a fusion strategy based on upsampling and feature splicing, which unifies the spatial resolution of features and effectively combines them to achieve deep fusion of multi-level features.

[0089] Specifically, the features Fn output by each convolutional layer are first upsampled to have the same spatial resolution. The upsampling process uses the bicubic interpolation method, which can maintain the smoothness and continuity of the features while improving the spatial resolution of the features, avoiding the block artifacts that may be caused by simple nearest neighbor interpolation. In particular, for deeper features, a larger multiple of upsampling is required to match the spatial size of the initial features. Subsequently, all upsampled features are concatenated in the channel dimension. This concatenation operation retains the complete information of the features of each layer. Finally, the concatenated features are channel fused and dimensionally reduced through a 3×3 convolutional layer to obtain the final target fused features.

[0090] Exemplarily, the feature fusion process can be expressed as: Where UP(a,b) represents upsampling a with a scaling factor of b, using bicubic interpolation. Decoder() represents the decoding process, where a 3×3 convolution is applied to restore the data to image form.

[0091] The above-mentioned feature fusion mechanism based on upsampling and splicing has significant advantages: first, the unified upsampling process ensures the alignment of features at each layer in the spatial dimension, which facilitates subsequent feature fusion; second, the splicing in the channel dimension maintains the complete information of features at each layer and avoids information loss.

[0092] S304: Decode the target fusion features to obtain a target fusion image.

[0093] After obtaining the target fusion feature, it needs to be converted into the final target fusion image. Since the target fusion feature is a high-dimensional representation in the feature space, it contains rich spatial structure and spectral information, and it needs to be mapped back to the image space through a decoding process to generate a fusion image with high spatial resolution and rich spectral information. To this end, the embodiment of the present application designs a feature decoding strategy, which realizes an effective conversion from feature space to image space through multi-level feature reconstruction and optimization.

[0094] Specifically, the decoding process first upsamples and transforms the target fusion features through a series of transposed convolutional layers. Each transposed convolutional layer not only improves the spatial resolution of the features, but also reorganizes the features through learnable convolutional kernels, gradually converting the abstract feature representation into specific image information. In particular, a batch normalization layer and a ReLU activation function are added after each transposed convolutional layer. This design can not only stabilize the distribution of features, but also introduce appropriate nonlinear transformation capabilities. In order to better maintain spatial details, a skip connection structure is also introduced in the decoding process to fuse the features of the corresponding layer in the encoding stage with the features of the current layer. This feature reuse mechanism helps to restore high-frequency detail information.

[0095] Based on the above embodiment, as an optional embodiment, please refer to Figure 2 The full-color image, multispectral image and prior image can be downsampled twice respectively to obtain three sets of different input images, and then the three sets of different input images are input into the remote sensing image spatial-spectral fusion network. The remote sensing image spatial-spectral fusion network can obtain target fusion features of three different scales. Finally, the target fusion features of the three different scales are spliced ​​and decoded to obtain the final fusion features.

[0096] In the actual remote sensing image fusion process, image features at different scales often contain complementary information. Since ground objects exhibit different feature patterns at different scales, feature extraction and fusion at only a single scale may lose some important spatial and spectral details. In order to more comprehensively capture multi-scale feature information, the embodiment of the present application adopts a multi-scale feature extraction strategy, which improves the integrity and accuracy of the fusion effect by downsampling the input image multiple times.

[0097] Specifically, the panchromatic image, multispectral image and prior image are first downsampled twice to generate three sets of input images with different spatial resolutions. The downsampling process is implemented using average pooling, because average pooling can maintain the main structural information of the image while reducing the image resolution, avoiding sudden loss of information or drastic changes in features. Through average pooling, each larger-scale feature map is converted into a smaller-scale feature map to form a multi-scale feature pyramid structure. This multi-scale processing method enables the network to focus on image features at different resolutions at the same time: the larger-scale feature map mainly contains the global structural information of the image, while the smaller-scale feature map retains more local detail information.

[0098] Subsequently, the three sets of input images of different scales are input into the remote sensing image spatial-spectral fusion network respectively. At each scale level, the network will extract and fuse the input images through the encoder, multiple residual feature extraction module, dual attention mechanism module and dense connection fusion module in sequence. This hierarchical processing method ensures that the network can effectively extract and integrate image features at various scales. In particular, the features of larger scales focus on the overall structure and semantic information of the image, the features of medium scales balance the local details and global structure, and the features of smaller scales focus more on capturing fine texture and edge information.

[0099] After completing the extraction and fusion of features at each scale, the network will obtain target fusion features of three different scales. In order to integrate these multi-scale feature information, it is necessary to first resample the feature maps of different scales to the same spatial resolution. This application uses a bicubic interpolation method to implement upsampling operations. This is because compared to nearest neighbor interpolation and bilinear interpolation, bicubic interpolation can produce smoother interpolation results and reduce the sawtooth effect and distortion during the resampling process. Subsequently, the resampled features are spliced ​​in the channel dimension. This splicing operation retains the complete information of the features of each scale. Finally, the spliced ​​multi-scale features are converted into the final fused image through the decoder, and the decoding process realizes the mapping conversion of features to the image domain through multi-layer convolution operations.

[0100] Based on the above embodiment, as an optional embodiment, please refer to Figure 6 , Figure 6 A schematic diagram of a loss function provided in an embodiment of the present application, wherein the target loss function includes: Spatial loss function, used to calculate the structural similarity between the target fused image and the sample full-color image; Spectral loss function, used to calculate the spectral angle mapping between the target fusion image and the sample multispectral image; A spatial-spectral combination loss function is used to calculate the difference between a first residual probability distribution and a second residual probability distribution, wherein the first residual probability distribution is the residual probability distribution between the target fused image and the sample panchromatic image, and the second residual probability distribution is the residual probability distribution between the sample multispectral image and the grayscale image corresponding to the sample multispectral image.

[0101] Among them, the spatial loss function can be derived from the structural similarity loss term, and the structural similarity loss term is obtained by the structural similarity index between the target fusion image and the sample full-color image.

[0102] In the process of spatial-spectral fusion of remote sensing images, maintaining the consistency of the spatial structure between the fused image and the panchromatic image is one of the key goals. Since the human visual system is particularly sensitive to image structural information, directly using the mean square error at the pixel level may not accurately reflect the visual quality of the image. To this end, the embodiment of the present application introduces a structural similarity loss term, which effectively constrains the reconstruction quality of the fused image in terms of spatial structure by evaluating the structural similarity index (SSIM) between the target fused image and the sample panchromatic image.

[0103] Specifically, the structural similarity index comprehensively evaluates the similarity of images by considering the brightness, contrast and structure of the images at the same time. First, the local statistical features between the target fused image IF and the sample full-color image IP are calculated, including the local mean, local standard deviation, and local covariance. These statistics are calculated within a local window to capture structural information of appropriate scales. Based on these statistics, the calculation of the SSIM index can be expressed as: In the formula, and Represent the mean of images x and y respectively, and Represent the standard deviation of images x and y respectively, represents the covariance between images x and y, and is a very small constant used to stabilize the output and avoid division by zero errors.

[0104] Among them, the spectral loss function can be derived from the spectral angle loss term, and the spectral angle loss term is obtained by the spectral angle mapping between the target fusion image and the sample multispectral image.

[0105] In the process of spatial-spectral fusion of remote sensing images, keeping the spectral characteristics of the fused image consistent with the original multispectral image is another key goal. Since the accuracy of spectral information directly affects subsequent remote sensing applications, such as object classification and change detection, it is not enough to focus only on the reconstruction of spatial structure. To this end, the embodiment of the present application introduces a spectral angle loss term, which effectively constrains the reconstruction quality of the fused image in the spectral dimension by calculating the spectral angle mapping (SAM) between the target fused image and the sample multispectral image.

[0106] Specifically, spectral angle mapping measures the similarity of two spectral vectors by calculating the angle between them. This measurement method is particularly sensitive to changes in spectral shape and is not affected by spectral intensity. For the target fused image IF and the sample multispectral image IMS, their spectral angle can be expressed as: Where x⋅y represents the dot product between vectors x and y. ||·|| represents the Euclidean norm of the vector (i.e., the length of the vector).

[0107] Among them, the spatial-spectral combination loss function can be derived from the probability distribution loss term, which is obtained by the KL divergence of the residual probability distribution between the target fusion image and the sample panchromatic image, and the residual probability distribution between the sample spectral image and the corresponding grayscale image of the sample multispectral image.

[0108] In the process of spatial-spectral fusion of remote sensing images, in addition to directly constraining the reconstruction quality of spatial structure and spectral features, it is also necessary to ensure that the statistical characteristics between the fusion result and the source image are consistent. Since the probability distribution characteristics of the image reflect its inherent statistical laws, directly constraining pixel values ​​or eigenvalues ​​may not be able to fully capture this statistical consistency. To this end, the embodiment of the present application introduces a probability distribution loss term, which constrains the rationality of the fusion process from a statistical perspective by calculating the probability distribution difference of the residual between the target fused image and the source image.

[0109] For example, the expression is as follows: Among them, p(x) represents the probability distribution function of the real information, and q(x) represents the probability distribution function of the fitted information. KL divergence measures the distribution gap between p(x) and q(x) by comparing the difference between them. The smaller the value, the closer the distribution of the fitted information is to that of the real information.

[0110] In the process of remote sensing image spatial-spectral fusion, in order to optimize both spatial resolution and spectral fidelity, a comprehensive loss function system needs to be designed. The embodiment of the present application proposes a joint loss function, including a spatial loss function L_spa, a spectral loss function L_spe, and a spatial-spectral combination loss function L_css, which comprehensively guides the network training process through these three complementary loss terms.

[0111] Exemplarily, the spatial loss function L_spa optimizes the spatial resolution by evaluating the structural similarity between the target fused image and the panchromatic image PAN. Considering that the panchromatic image is single-channel and the target fused image is multi-channel, the PAN image needs to be expanded in the channel dimension so that it has the same number of channels as the target fused image. The calculation formula of the spatial loss function is: In the formula, the stack operation expands the single-channel PAN image to the same number of channels as the target fused image. This loss term ensures that the fusion result has similar spatial structural characteristics as the full-color image in each band.

[0112] For example, the spectral loss function L_spe focuses on the spectral consistency between the target fused image and the original multispectral image MS. The spectral difference is measured by calculating the spectral angle mapping between the two images, and the calculation formula is: This loss term ensures that the fusion result can maintain the spectral characteristics of the original multispectral image.

[0113] Furthermore, in order to establish the connection between spatial features and spectral features, the spatial-spectral combination loss function L_css is introduced. In order to construct a loss function that can comprehensively compare the target fusion image with PAN and MS, the KL divergence is introduced into the loss function based on the following assumptions: a. A full-color image with the same resolution as the MS image can be obtained by converting the MS image into a grayscale image; b. Grayscale images contain spatial texture information of the source image; c. The differences between MS images and PAN images at different spatial resolutions should follow a similar distribution.

[0114] Based on the above assumptions, we first use the grayscale degradation network to Image Generate Grayscale Image The network uses a channel-based attention mechanism through convolutional layers and global average pooling to highlight relevant channels, suppress irrelevant channels, and compress multi-channel inputs into single-channel results, thereby completing grayscale conversion. The outputs are then stacked back to their original format, please refer to Figure 7 , Figure 7 A schematic diagram of a grayscale degradation network process provided by an embodiment of the present application. Using the softmax function in MS images and grayscale images A residual probability distribution graph is constructed between . This residual distribution is used to fit the residual probability distribution between the target fusion image and PAN. Finally, the KL divergence between these two probability distributions is minimized.

[0115] Exemplarily, the spatial-spectral combination loss function has the following specific formula: In the formula, gray() represents the grayscale degradation network, is achieved by degenerating the network The resulting grayscale image.

[0116] The final joint loss function is a weighted combination of these three loss terms: In the formula, the values ​​of α, β and γ are determined by experiments in short-cycle training mode. By using SSIM, SAM and KL divergence as loss functions, the network training process can be effectively optimized, thereby improving the network's fusion performance.

[0117] The above-mentioned training strategy based on multiple losses ensures the reconstruction quality of high-resolution details through spatial loss; secondly, spectral loss ensures the accurate preservation of spectral information; finally, the spatial-spectral joint loss establishes the association between spatial and spectral features, providing a more comprehensive optimization goal. Experimental results show that this multi-dimensional loss function design can effectively guide the training of the network, so that the fusion results have both high spatial resolution and accurate spectral features. Especially when dealing with complex scenes, it shows excellent balancing ability. The generated fusion image not only maintains clear spatial details, but also accurately retains spectral information, providing high-quality data support for the practical application of remote sensing images.

[0118] Furthermore, in terms of the selection of training and test data, the embodiment of the present application uses the WorldView-2 (WV-2) and Gaofen series (GF) satellite datasets to evaluate the performance of the proposed fusion method. Specifically, the WV-2 dataset is used for training and verification of the network, while the GF series datasets (including GF1, GF1B, GF1C, GF1D, GF2, and GF6) are used to evaluate the generalization ability of the network. Since the WV-2 dataset contains more spectral bands than the GF-1 dataset, four bands (bands 2, 3, 4, and 7) in the WV-2 dataset can be screened according to the spectral coverage to ensure that they are consistent with the spectral range of the GF dataset. This choice enables the weights trained on the WV-2 dataset to be effectively migrated and applicable to the GF series data, thereby improving the adaptability and generalization ability of the model on different satellite data.

[0119] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of a self-supervised remote sensing image spatial-spectral fusion device based on a priori image constraints provided in an embodiment of the present application. The self-supervised remote sensing image spatial-spectral fusion device based on a priori image constraints comprises: An initial image acquisition module, used to acquire full-color images and multispectral images; A priori image generation module, used to generate a priori image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, wherein the priori image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network on the panchromatic image and the multispectral image; A target image fusion module is used to input the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fused image output by the remote sensing image spatial-spectral fusion network; Among them, the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image and the sample multispectral image.

[0120] On the basis of the above embodiment, as an optional embodiment, the prior image generation module is further used to inject the spatial information in the panchromatic image into the multispectral image to obtain the prior image. .

[0121] On the basis of the above embodiments, as an optional embodiment, the target image fusion module is also used to perform feature extraction on the full-color image, the multispectral image and the prior image through the residual module to obtain deep features; extract spatial features and spectral features in the deep features through the dual attention module, and extract and splice the spatial features and spectral features to obtain spliced ​​features; extract and fuse multi-level features in the spliced ​​features through the dense fusion module to obtain target fusion features; decode the target fusion features to obtain a target fusion image.

[0122] On the basis of the above embodiments, as an optional embodiment, the target image fusion module is further used to encode the full-color image, the multispectral image and the prior image through an encoder respectively to obtain encoding features; splice the encoding features corresponding to the full-color image, the multispectral image and the prior image in pairs to obtain initial features; process the initial features through the convolution layer, the normalization layer and the activation function layer to obtain depth features.

[0123] Based on the above embodiment, as an optional embodiment, the target image fusion module is further used to perform global maximum pooling operation and global average pooling operation on the depth feature through the spatial attention submodule, and obtain a spatial attention map through convolution operation, and multiply the spatial attention map with the depth feature to obtain a spatial feature; perform a global average pooling operation on the depth feature through the spectral attention submodule, and obtain a spectral attention map through a convolution layer, and multiply the spectral attention map with the depth feature to obtain a spectral feature; splice the spatial feature and the spectral feature to obtain a spliced ​​feature. .

[0124] Based on the above embodiment, as an optional embodiment, the target image fusion module is also used to pass the stitching features through each of the convolutional layers in sequence to obtain multi-level features; and perform a stitching operation on the multi-level features to obtain a target fusion feature.

[0125] Fig. 9A schematic diagram of a physical structure of an electronic device is provided for an embodiment of the present application, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints.

[0126] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0127] On the other hand, the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints provided by the above-mentioned methods.

[0128] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints provided by the above-mentioned methods.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present application. Those of ordinary skill in the art may understand and implement it without creative labor.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints, characterized in that: include: Acquire panchromatic and multispectral images; Based on the spatial information of the panchromatic image and the spectral information of the multispectral image, a priori image is generated, wherein the priori image is used to constrain the fusion process of the panchromatic image and the multispectral image by the remote sensing image spatial-spectral fusion network; Inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain the remote sensing image spatial-spectral fusion network output target fused image; Among them, the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image and the sample multispectral image.

2. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 1, characterized in that: The generating of the prior image based on the spatial information of the panchromatic image and the spectral information of the multispectral image comprises: The spatial information in the panchromatic image is injected into the multispectral image to obtain a priori image.

3. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 1, characterized in that: The remote sensing image space-spectrum fusion network includes a residual module, a dual attention module and a dense fusion module; The step of inputting the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain the remote sensing image spatial-spectral fusion network output target fused image comprises: Performing feature extraction on the panchromatic image, the multispectral image and the prior image through the residual module to obtain a depth feature; Extracting spatial features and spectral features from the deep features through the dual attention module, and extracting and splicing the spatial features and spectral features to obtain spliced ​​features; Extracting and fusing multi-level features in the splicing features through the dense fusion module to obtain target fusion features; The target fusion feature is decoded to obtain a target fusion image.

4. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 3 is characterized in that: The residual module includes an encoder, a convolution layer, a normalization layer and an activation function layer; The extracting features of the panchromatic image, the multispectral image and the prior image by the residual module to obtain the depth feature includes: Encode the panchromatic image, the multispectral image, and the prior image respectively through encoders to obtain encoding features; The encoding features corresponding to the panchromatic image, the multispectral image and the prior image are concatenated in pairs to obtain initial features; The initial features are processed through the convolution layer, the normalization layer and the activation function layer to obtain deep features.

5. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 3, characterized in that: The dual attention module includes a spatial attention submodule and a spectral attention submodule; The step of extracting the spatial features and the spectral features in the deep features by the dual attention module and splicing the spatial features and the spectral features to obtain spliced ​​features includes: Performing a global maximum pooling operation and a global average pooling operation on the deep features through the spatial attention submodule, and obtaining a spatial attention map through a convolution operation, and multiplying the spatial attention map with the deep features to obtain spatial features; Performing a global average pooling operation on the deep features through the spectral attention submodule, obtaining a spectral attention map through a convolution layer, and multiplying the spectral attention map with the deep features to obtain spectral features; The spatial feature and the spectral feature are spliced ​​together to obtain a spliced ​​feature.

6. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 3, characterized in that: The dense fusion module includes a plurality of convolutional layers; The step of extracting and fusing the multi-level features in the splicing features through the dense fusion module to obtain the target fusion features includes: Passing the concatenated features through each of the convolutional layers in sequence to obtain multi-level features; The multi-level features are concatenated to obtain target fusion features.

7. The method for self-supervised remote sensing image spatial-spectral fusion based on prior image constraints according to claim 1, characterized in that: The objective loss function includes: A spatial loss function, used to calculate the structural similarity between the target fused image and the sample full-color image; A spectral loss function, used to calculate a spectral angle mapping between the target fused image and the sample multispectral image; A spatial-spectral combination loss function is used to calculate the difference between a first residual probability distribution and a second residual probability distribution, wherein the first residual probability distribution is the residual probability distribution between the target fused image and the sample panchromatic image, and the second residual probability distribution is the residual probability distribution between the sample multispectral image and the grayscale image corresponding to the sample multispectral image.

8. A self-supervised remote sensing image spatial-spectral fusion device based on prior image constraints, characterized in that: include: An initial image acquisition module, used to acquire full-color images and multispectral images; A priori image generation module, used to generate a priori image based on the spatial information of the panchromatic image and the spectral information of the multispectral image, wherein the priori image is used to constrain the fusion process of the remote sensing image spatial-spectral fusion network on the panchromatic image and the multispectral image; A target image fusion module is used to input the panchromatic image, the multispectral image and the prior image into the remote sensing image spatial-spectral fusion network to obtain a target fused image output by the remote sensing image spatial-spectral fusion network; Among them, the remote sensing image spatial-spectral fusion network is trained by self-supervision based on the target loss function, and the target loss function is constructed by calculating the probability distribution difference of spatial loss and spectral loss between the target fusion image, the sample panchromatic image and the sample multispectral image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the self-supervised remote sensing image spatial-spectral fusion method based on prior image constraints as described in any one of claims 1 to 7 is implemented.

Images