A method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network

Abstract

The present invention discloses a method for pan-sharpening of multispectral remote sensing images based on a frequency domain decomposition network. The method constructs a frequency domain decomposition and domain generalization network, which includes three core modules: a dynamic convolution low-frequency extraction module, a high-frequency retention module, and a spectrum enhancement module. A domain generalization training strategy is proposed, which adopts a generative adversarial network structure and introduces an adversarial loss function. By intentionally misassigning domain labels, the generator is guided to achieve better cross-domain generalization capabilities under the supervision of the discriminator. The present invention effectively captures information of different scales, improves the quality of fused images, and realizes accurate fusion of invisible satellite data.

Description

Technical Field

[0001] The invention belongs to the technical field of image fusion, and in particular relates to a multi-spectral remote sensing image full color sharpening method based on a frequency domain decomposition network. Background Art

[0002] With the widespread application of remote sensing images in the scientific field, the demand for high-resolution multispectral images is growing. However, due to the inherent physical limitations of satellite imaging systems, only high-resolution panchromatic images and low-resolution multispectral images can be obtained. In order to meet the needs of practical applications, panchromatic sharpening technology is used to fuse these two types of images to generate high-resolution multispectral images.

[0003] Existing pan-sharpening methods are mainly divided into two categories: traditional methods and deep learning-based methods. Traditional methods can be divided into three categories, namely component replacement methods, multi-resolution analysis methods and variational optimization methods. In recent years, methods based on deep learning have developed rapidly. The pan-sharpening method based on convolutional neural networks described in the paper Giuseppe, Masi, et al. "Pansharpening byConvolutional Neural Networks." Remote Sensing 8.7(2016):594. achieves efficient pan-sharpening effect by using only three convolutional layers. Subsequently, a large number of deep neural network structures and fusion strategies have been proposed to further improve the performance of pan-sharpening.

[0004] Although deep learning-based methods have made significant progress in pan-sharpening tasks and their performance is excellent on the same satellite dataset, they still have the problem of insufficient adaptability when dealing with different satellite images and cross-domain applications. Therefore, further technical exploration and improvement are still needed to address the domain generalization problem of different satellite data. For most existing deep learning-based pan-sharpening methods, their network models are usually trained and tested on images from the same satellite. However, when the training and test data come from different satellites, the pan-sharpening performance will drop significantly. This limitation is mainly due to the fact that traditional deep learning networks tend to learn specific spectral patterns in the training domain and have difficulty adapting to the unique characteristics of new satellite images. Therefore, existing methods have poor adaptability when dealing with the domain shift problem of different satellite data, which limits their practical application. To address this problem, it is urgent to develop a pan-sharpening network that can effectively deal with domain shift. Once a generalized model is developed, it can be directly used in satellite-borne equipment without retraining for specific satellite data. In addition, since satellite images change over time, a generalized model is particularly critical for real-time pan-sharpening, which can significantly improve image resolution to meet the needs of immediate applications. Summary of the invention

[0005] The purpose of the present invention is to provide a multispectral remote sensing image panchromatic sharpening method based on a frequency domain decomposition network, which can effectively capture information of different scales, improve the quality of fused images, and achieve accurate fusion of invisible satellite data.

[0006] In order to achieve the purpose of the present invention, the present invention provides a multispectral remote sensing image panchromatic sharpening method based on a frequency domain decomposition network, comprising the following steps:

[0007] Step 1: giving satellite data different domain labels to obtain satellite data with domain labels and satellite data test images that are invisible during training; inputting the true value images in the satellite data with domain labels into the discriminator for training to obtain a trained discriminator;

[0008] Step 2: extracting the overall features of the satellite data with domain labels to obtain preliminary features, dividing the preliminary features into bands and performing discrete wavelet transformation to obtain high-frequency and low-frequency components of the corresponding sub-bands; performing discrete wavelet transformation on the high-resolution panchromatic image and the low-resolution panchromatic image in the satellite data with domain labels to obtain corresponding high-frequency and low-frequency components;

[0009] Step 3, the high-frequency components of the high-resolution full-color image and the preliminary features of the sub-bands are retained at high frequencies to obtain spatial features; the low-frequency components of the low-resolution full-color image and the preliminary features of the sub-bands are extracted through dynamic convolution low frequencies to obtain spectral features; the spatial features and the spectral features are transformed through inverse discrete wavelet transform and channel splicing to obtain complete fusion features;

[0010] Step 4: spectrally enhance the complete fusion features to generate a high-resolution multispectral image, and input the high-resolution multispectral image into the trained discriminator to calculate the adversarial loss optimization generator to obtain a trained generator;

[0011] Step 5: input the satellite data test image that is invisible during training into the trained generator to obtain a fused image; and evaluate the fusion effect of the fused image through quantitative evaluation indicators.

[0012] Compared with the prior art, the present invention has the following significant improvements:

[0013] (1) Different from the traditional deep learning method that uses a single resolution framework, the present invention uses wavelet decomposition technology to decompose multispectral images and high-resolution panchromatic images into high-frequency and low-frequency components of different scales, and learns them separately; this strategy can effectively capture information of different scales under the condition of reduced resolution training and meet the performance requirements of testing at the original resolution;

[0014] (2) The present invention does not directly perform training on the original domain of the image, but decomposes the multispectral image and the high-resolution panchromatic image into high-frequency and low-frequency parts through wavelet decomposition, thereby achieving effective separation of spectral information and spatial information; this method can reasonably balance spectral and spatial information in the feature extraction process, prevent the fusion interference problem of the two in the high-resolution multispectral results, and thus improve the quality of the fused image;

[0015] (3) This paper introduces a domain-invariant feature learning strategy and designs a new adversarial loss function. By intentionally assigning incorrect domain labels, the generator is guided to perform cross-domain feature generalization learning under the supervision of the discriminator. This method can significantly enhance the generalization performance of the model for different satellite data, achieve accurate fusion of unseen satellite data, and meet the needs of cross-domain applications.

[0016] In order to more clearly illustrate the functional characteristics and structural parameters of the present invention, further description is given below in conjunction with the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 is a flow chart of the method of the present invention;

[0019] Figure 2 It is the overall network structure diagram of the present invention;

[0020] Figure 3 It is a structural diagram of the spatial feature extraction step of the present invention;

[0021] Figure 4 It is a structural diagram of the spectral feature extraction step of the present invention;

[0022] Figure 5 It is a structure diagram of the spectrum enhancement step of the present invention;

[0023] FIG6( a ) is a low-pass full-color image of the present invention;

[0024] FIG6( b) is a low-resolution multi-spectral image of the present invention;

[0025] FIG6( c ) is a full-color image of the present invention;

[0026] FIG6( d ) is a diagram showing the image fusion result of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] The present invention is a multi-spectral remote sensing image panchromatic sharpening method based on frequency domain decomposition network, combined with Figure 1 and Figure 2 , including the following steps:

[0029] Step 1: giving satellite data different domain labels to obtain satellite data with domain labels and satellite data test images that are invisible during training; inputting the true value images in the satellite data with domain labels into the discriminator for training to obtain a trained discriminator;

[0030] The cross entropy loss function is calculated using the domain labels of the ground truth images in the satellite data with domain labels and the predicted domain labels. Train the discriminator:

[0031] ;

[0032] in, represents the domain label of the true value image, Indicates the number of label categories, Indicates The true labels of each field are encoded using one-hot encoding. Indicates that the input image belongs to The predicted probability of each field;

[0033] The learning rate of the discriminator is set to 0.001, and the optimization algorithm uses the Adam optimizer. The training process is carried out for 50 epochs. The discriminator is trained until it can accurately identify the domain label of the input image, and the optimization of the discriminator is completed. After the training is completed, the weight of the discriminator is frozen to ensure the stability of the generator during training and obtain a trained discriminator.

[0034] Step 2: extracting the overall features of the satellite data with domain labels to obtain preliminary features, dividing the preliminary features into bands and performing discrete wavelet transformation to obtain high-frequency and low-frequency components of the corresponding sub-bands; performing discrete wavelet transformation on the high-resolution panchromatic image and the low-resolution panchromatic image in the satellite data with domain labels to obtain corresponding high-frequency and low-frequency components;

[0035] The step 2 is specifically as follows: overall feature extraction and component transformation: the satellite data with domain labels are divided into low-resolution multispectral images, high-resolution panchromatic images and true value images, and the high-resolution panchromatic images are downsampled and upsampled to obtain low-resolution panchromatic images; the low-resolution multispectral images, the high-resolution panchromatic images, the low-resolution panchromatic images, and the erroneous domain labels are spliced ​​together in the channel dimension and input into the overall feature extraction network to obtain preliminary features; the preliminary feature division obtains preliminary features of sub-bands, and the preliminary features of the sub-bands, the high-resolution panchromatic images and the low-resolution panchromatic images are subjected to discrete wavelet transformation to obtain corresponding high-frequency and low-frequency features, and the high-frequency and low-frequency features are input into the subsequent network for learning.

[0036] The preliminary characteristics The formula is as follows:

[0037] ;

[0038] in, represents a low-resolution multispectral image, represents a high-resolution full-color image, represents a low-resolution full-color image, A field label representing an error. Represents the embedding technology, which maps the domain label to a feature vector. represents the overall feature extraction network, which consists of four ResBlocks, and the number of channels of each ResBlock is 64;

[0039] The high-frequency and low-frequency features are input into the subsequent network for learning. The formula is as follows:

[0040] ;

[0041] ;

[0042] ;

[0043] in, Represents the preliminary characteristics of the sub-bands, stands for discrete wavelet transform, , They represent the low-frequency and high-frequency components of the corresponding image respectively.

[0044] Step 3, the high-frequency components of the high-resolution full-color image and the preliminary features of the sub-bands are retained at high frequencies to obtain spatial features; the low-frequency components of the low-resolution full-color image and the preliminary features of the sub-bands are extracted through dynamic convolution low frequencies to obtain spectral features; the spatial features and the spectral features are transformed through inverse discrete wavelet transform and channel splicing to obtain complete fusion features;

[0045] Combination Figure 3 , the step 3 specifically comprises the following steps:

[0046] 3-1. Spatial feature extraction: The high-frequency components of the high-resolution full-color image and the preliminary features of the sub-bands are spliced ​​in the channel dimension to obtain splicing features, and the splicing features are input into two Convolutional layer, and add activation function in the middle to extract the final high-frequency features; use learning parameters , performing feature balance learning on the high-frequency features and the high-frequency components of the preliminary features of the sub-bands, and controlling their contributions to obtain spatial features;

[0047] The splicing feature for:

[0048] ;

[0049] in, Represents the concatenation of features in the channel dimension;

[0050] The high frequency characteristics for:

[0051] ;

[0052] in, Represents the LeakyReLU activation function;

[0053] The spatial characteristics are:

[0054] ;

[0055] in, Representative The spatial characteristics of the spectral bands.

[0056] Combination Figure 4 , 3-2, spectral feature extraction: the low-resolution full-color image and the preliminary feature low-frequency component of the sub-band are subtracted to obtain differential features; the differential features are input into the feature extraction module to generate dynamic convolution; the preliminary feature low-frequency component of the sub-band is used to obtain dynamic features by dynamic convolution; the dynamic features and the preliminary feature low-frequency component of the sub-band are subtracted to obtain the first Spectral characteristics of each band;

[0057] The differential feature for:

[0058] ;

[0059] The dynamic convolution for:

[0060] ;

[0061] in, Represents a The convolutional layer, Represents a residual module consisting of two layers;

[0062] The dynamic characteristics for:

[0063] ;

[0064] in, Represents matrix element-by-element multiplication;

[0065] No. Spectral characteristics of each band for:

[0066] .

[0067] Step 4: spectrally enhance the complete fusion features to generate a high-resolution multispectral image, and input the high-resolution multispectral image into the trained discriminator to calculate the adversarial loss optimization generator to obtain a trained generator;

[0068] Combination Figure 5 , the step 4 specifically comprises the following steps:

[0069] 4-1. Spectral enhancement: The feature space features and spectral features extracted in step 3-1 and step 3-2 are fused through inverse discrete wavelet transform to reconstruct the final reconstructed features; the reconstructed features are spliced ​​on the channel to obtain a complete fusion feature;

[0070] The complete fusion feature is spliced ​​with the low-resolution multispectral image in the channel dimension to form an initial input feature; the initial input feature is processed through a convolution layer and an activation function to obtain a preliminary feature; the preliminary feature is input into a convolution layer and extracted by an activation function to obtain a deep feature; the deep feature is subjected to an activation function to generate an attention weight of a channel; the attention weight is multiplied by the preliminary feature to obtain a weighted feature map; the initial input feature, the weighted feature map, and the low-resolution multispectral image matrix are added to obtain a fusion result;

[0071] The initial input features for:

[0072] ;

[0073] in, represents the complete fusion feature;

[0074] The preliminary characteristics for:

[0075] ;

[0076] in, Represents the ReLU activation function;

[0077] The deep features for:

[0078] ;

[0079] The attention weight for:

[0080] ;

[0081] in, Represents the sigmoid activation function;

[0082] The weighted feature map for:

[0083] ;

[0084] The fusion result for:

[0085] .

[0086] 4-2. New loss training: input the fusion result into the trained discriminator to calculate the adversarial domain classification loss; use the fusion result and the true value image to calculate the full color sharpening loss, the full color sharpening loss and the adversarial domain classification loss are weighted to obtain the overall loss function, and the overall loss function is used to optimize the generator to obtain a trained generator;

[0087] The adversarial domain classification loss for:

[0088] ;

[0089] in, represents the generator network, i.e. the pan-sharpening network, Represents the first prediction of the trained discriminator The probability of a domain label;.

[0090] The pan-sharpening loss for:

[0091] ;

[0092] in, represents the fusion result, Represents the true value image.

[0093] The overall loss function for:

[0094] ;

[0095] in, Represents the weight for balancing the contribution of pan-sharpening loss and domain loss.

[0096] In order to ensure stable training of the generator, during the generator training phase, the learning rate is set to 0.0001 and the batch size is set to 32; the optimization algorithm also uses the Adam optimizer, and the weight decay coefficient is set to 1e-5. The total number of epochs for generator training is 100; through the above training process, the effective coordination of the generator and the discriminator is achieved, and a trained generator is obtained.

[0097] Step 5: input the satellite data test image that is invisible during training into the trained generator to obtain a fused image; and evaluate the fusion effect of the fused image through quantitative evaluation indicators.

[0098] Traversing the above steps, we get the simulation results:

[0099] Combined with Figure 6 (a), a high-resolution panchromatic image is an image generated by collecting data from a sensor in a wide spectral range, which usually covers visible light and part of the near-infrared band. It is a single-band grayscale image. Unlike a multispectral image, a high-resolution panchromatic image does not distinguish between different bands, but integrates light signals of all wavelengths into one band.

[0100] Combined with Figure 6 (b), a multispectral image is an image that records a target object or scene by capturing information in multiple bands; each band corresponds to a different range of electromagnetic wavelengths, thereby providing rich spectral information for distinguishing the characteristics of an object;

[0101] Combined with Figure 6 (c), the high-resolution panchromatic image is low-pass filtered to obtain a low-resolution panchromatic image;

[0102] Combined with Figure 6(d), a high-resolution multispectral image is formed by fusing a high-resolution panchromatic image and a low-resolution hyperspectral image.

[0103] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0104] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for panchromatic sharpening of multispectral remote sensing images based on frequency domain decomposition network, characterized in that: The following steps are involved: Step 1: Give satellite data different domain labels to obtain satellite data with domain labels and satellite data test images that are invisible during training; Inputting the true value image in the satellite data with domain labels into the discriminator for training to obtain a trained discriminator; Step 2: extracting the overall features of the satellite data with domain labels to obtain preliminary features, dividing the preliminary features into bands and performing discrete wavelet transformation to obtain high-frequency and low-frequency components of the corresponding sub-bands; performing discrete wavelet transformation on the high-resolution panchromatic image and the low-resolution panchromatic image in the satellite data with domain labels to obtain corresponding high-frequency and low-frequency components; Step 3, the high-frequency components of the high-resolution full-color image and the preliminary features of the sub-bands are retained at high frequencies to obtain spatial features; the low-frequency components of the low-resolution full-color image and the preliminary features of the sub-bands are extracted through dynamic convolution low frequencies to obtain spectral features; the spatial features and the spectral features are transformed through inverse discrete wavelet transform and channel splicing to obtain complete fusion features; 3-1. Spatial feature extraction: The high-frequency components of the high-resolution full-color image and the preliminary features of the sub-bands are spliced ​​in the channel dimension to obtain splicing features, and the splicing features are input into two Convolutional layer, and add activation function in the middle to extract the final high-frequency features; Using learning parameters , performing feature balance learning on the high-frequency features and the high-frequency components of the preliminary features of the sub-bands, and controlling their contributions to obtain spatial features; 3-2. Spectral feature extraction: subtract the low-resolution full-color image from the low-frequency component of the preliminary feature of the sub-band to obtain a differential feature; input the differential feature into a feature extraction module to generate a dynamic convolution; use the dynamic convolution to obtain a dynamic feature from the low-frequency component of the preliminary feature of the sub-band; subtract the dynamic feature from the low-frequency component of the preliminary feature of the sub-band to obtain the first Spectral characteristics of each band; Step 4: spectrally enhance the complete fusion features to generate a high-resolution multispectral image, and input the high-resolution multispectral image into the trained discriminator to calculate the adversarial loss optimization generator to obtain a trained generator; Step 5: input the satellite data test image that is invisible during training into the trained generator to obtain a fused image; and evaluate the fusion effect of the fused image through quantitative evaluation indicators.

2. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 1, characterized in that: The cross entropy loss function is calculated using the domain labels of the ground truth images in the satellite data with domain labels and the predicted domain labels. Train the discriminator in step 1: ； in, represents the domain label of the true value image, Indicates the number of label categories, Indicates The true labels of each field are encoded using one-hot encoding. Indicates that the input image belongs to The predicted probability of a field.

3. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 2, characterized in that: The step 2 is specifically as follows: Overall feature extraction and component transformation: the satellite data with domain labels are divided into low-resolution multispectral images, high-resolution panchromatic images and true value images, and the high-resolution panchromatic images are downsampled and upsampled to obtain low-resolution panchromatic images; the low-resolution multispectral images, the high-resolution panchromatic images, the low-resolution panchromatic images, and the erroneous domain labels are spliced ​​together in the channel dimension and input into the overall feature extraction network to obtain preliminary features; the preliminary feature division obtains preliminary features of sub-bands, and the preliminary features of the sub-bands, the high-resolution panchromatic images and the low-resolution panchromatic images are subjected to discrete wavelet transformation to obtain corresponding high-frequency and low-frequency features, and the high-frequency and low-frequency features are input into the subsequent network for learning.

4. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 3, characterized in that: The preliminary features in step 2 for: ； in, represents a low-resolution multispectral image, represents a high-resolution full-color image, represents a low-resolution full-color image, A field label representing an error. Represents the embedding technology, which maps the domain label to a feature vector. represents the overall feature extraction network, which consists of four ResBlocks, and the number of channels of each ResBlock is 64; The high-frequency and low-frequency features are input into the subsequent network for learning. The formula is as follows: ； ； ； in, Represents the preliminary characteristics of the sub-bands, stands for discrete wavelet transform, , They represent the low-frequency and high-frequency components of the corresponding image respectively.

5. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 4, characterized in that: In step 3-1 The splicing feature for: ； in, Represents the concatenation of features in the channel dimension; The high frequency characteristics for: ； in, Represents the LeakyReLU activation function; The spatial characteristics are: ； in, Representative The spatial characteristics of the spectral bands.

6. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 5, characterized in that: In step 3-2 The differential feature for: ； The dynamic convolution for: ； in, Represents a The convolutional layer, Represents a residual module consisting of two layers; The dynamic characteristics for: ； in, Represents matrix element-by-element multiplication; No. Spectral characteristics of each band for: 。 7. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 6, characterized in that: The step 4 specifically comprises the following steps: 4-1. Spectral enhancement: The feature space features and spectral features extracted in step 3-1 and step 3-2 are fused through inverse discrete wavelet transform to reconstruct the final reconstructed features; the reconstructed features are spliced ​​on the channel to obtain a complete fusion feature; The complete fusion feature is spliced ​​with the low-resolution multispectral image in the channel dimension to form an initial input feature; the initial input feature is processed through a convolution layer and an activation function to obtain a preliminary feature; the preliminary feature is input into a convolution layer and extracted by an activation function to obtain a deep feature; the deep feature is subjected to an activation function to generate an attention weight of a channel; the attention weight is multiplied by the preliminary feature to obtain a weighted feature map; the initial input feature, the weighted feature map, and the low-resolution multispectral image matrix are added to obtain a fusion result; 4-2. New loss training: input the fusion result into the trained discriminator to calculate the adversarial domain classification loss; use the fusion result and the true value image to calculate the full-color sharpening loss, and obtain the overall loss function by weighted sum of the full-color sharpening loss and the adversarial domain classification loss. Use the overall loss function to optimize the generator to obtain a trained generator.

8. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 7, characterized in that: In step 4-1 The initial input features for: ； in, represents the complete fusion feature; The preliminary characteristics for: ； in, Represents the ReLU activation function; The deep features for: ； The attention weight for: ； in, Represents the sigmoid activation function; The weighted feature map for: ； The fusion result for: 。 9. The method for pan-sharpening of multispectral remote sensing images based on frequency domain decomposition network according to claim 8, characterized in that: In step 4-2 The adversarial domain classification loss for: ； in, represents the generator network, i.e. the pan-sharpening network, Represents the first prediction of the trained discriminator The probability of domain labels; The pan-sharpening loss for: ； in, represents the fusion result, represents the true value image; The overall loss function for: ； in, Represents the weight for balancing the contribution of pan-sharpening loss and domain loss.

Images