Hyperspectral image sharpening method based on dynamic frequency enhancement

Abstract

The invention discloses a hyperspectral image sharpening method based on dynamic frequency enhancement, and the method comprises the steps: reading a hyperspectral image data set, and carrying out the processing of a hyperspectral image in the data set, and obtaining a full-color image and a low-resolution hyperspectral image; constructing a dynamic frequency enhanced panchromatic sharpening network, wherein the panchromatic sharpening network comprises a primary fusion network, a dynamic high and low frequency separation module and an adaptive frequency characteristic enhancement module; training a panchromatic sharpening network with enhanced dynamic frequency; and inputting the hyperspectral image into the trained panchromatic sharpening network to obtain a sharpened hyperspectral image. According to the dynamic frequency enhancement network provided by the invention, the frequency characteristics can be well enhanced in the frequency domain, the generalization ability is very strong, and the network parameters can be dynamically adjusted according to different characteristic graphs to realize dynamic enhancement.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image processing, and in particular to a hyperspectral image sharpening method based on dynamic frequency enhancement. Background Art

[0002] With the rapid development of remote sensing technology, remote sensing images have been widely used in fields such as change detection and disaster monitoring. However, due to the limitations of the imaging characteristics of sensors, there is currently no sensor that can simultaneously obtain hyperspectral images with high spatial resolution. Therefore, in practical applications, remote sensing sensors usually obtain two extreme types of images: one is an image with high spectral information but low spatial resolution, and the other is a panchromatic image with high spatial resolution but lack of spectral details. In order to solve this problem, panchromatic sharpening technology is widely used, which generates hyperspectral images with high spatial resolution by fusing the characteristics of hyperspectral images and panchromatic images.

[0003] Existing panchromatic sharpening methods can be mainly divided into two categories. One is the traditional method based on a priori mathematical models, such as component replacement, multi-scale analysis and Bayesian analysis. These methods usually face the problems of insufficient precision and large amount of calculation, resulting in unsatisfactory final fusion effect. The other is the technology based on deep learning, which usually learns the mapping relationship between input and output by designing various neural networks. Traditional mathematical calculation methods are often difficult to effectively capture the frequency characteristics of different bands in the fusion of hyperspectral images and panchromatic images, especially the significant difference between high-frequency and low-frequency responses. Although deep learning methods can automatically learn features, most network designs still rely on experience, mainly focusing on the spatial domain, and ignoring the inherent frequency characteristics of hyperspectral images. This design method is prone to cause the significance of frequency differences when processing input images, thereby affecting the fusion quality and causing distortion of spectral data. Therefore, how to better consider frequency characteristics in neural network design is still a key issue to be solved in the field of panchromatic sharpening. Summary of the invention

[0004] In order to overcome the inadequate consideration of the frequency domain by existing neural networks, the present invention provides a hyperspectral image sharpening method based on dynamic frequency enhancement. This method can fully consider the characteristics of the frequency domain when extracting features by the neural network, and can dynamically enhance the global and local frequency features for different input images.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A hyperspectral image sharpening method based on dynamic frequency enhancement, comprising:

[0007] Read the hyperspectral image data set, and process the hyperspectral images in the data set to obtain full-color images and low-resolution hyperspectral images;

[0008] Constructing a dynamic frequency enhanced full-color sharpening network, the full-color sharpening network includes a primary fusion network, a dynamic high-low frequency separation module and an adaptive frequency feature enhancement module;

[0009] Training a pan-sharpening network for dynamic frequency enhancement;

[0010] The hyperspectral image is input into the trained panchromatic sharpening network to obtain the sharpened hyperspectral image.

[0011] Furthermore, the hyperspectral images in the data set are processed to obtain a full-color image and a low-resolution hyperspectral image, specifically: the visible light bands contained in the spectral image are averaged to obtain a full-color image;

[0012] The hyperspectral image is Gaussian filtered and then downsampled by s times to obtain a low-resolution hyperspectral image, which is then divided into training set, validation set and test set by insufficient slicing.

[0013] Furthermore, the resolution relationship between the low-resolution hyperspectral image and the hyperspectral image is h=h 1 ×r,w=w 1 ×r, where r is the resolution multiple of the hyperspectral image and the panchromatic image.

[0014] Furthermore, the pan-sharpening network includes a primary fusion network, a dynamic high-low frequency separation module and an adaptive frequency feature enhancement module, and its processing flow is as follows:

[0015] An upsampled image of the hyperspectral image is obtained, and the upsampled image and the panchromatic image are spliced ​​and input into the primary fusion network to obtain the primary fusion features;

[0016] Input the primary fusion features into the dynamic high-low frequency separation module to obtain high-frequency features and low-frequency features;

[0017] The high-frequency features and low-frequency features are respectively input into the adaptive frequency feature enhancement module to obtain the enhanced high-frequency features and low-frequency features. The enhanced high-frequency features and low-frequency features are concatenated and then convolved for dimensionality reduction. They are then added to the upsampled image to obtain a sharpened hyperspectral image.

[0018] Furthermore, the primary fusion network includes a convolutional layer and n residual convolutional layers.

[0019] Furthermore, the processing process of the dynamic high and low frequency separation module is as follows:

[0020] The primary fusion features are passed through a parameter prediction network to obtain predicted frequency domain Gaussian mask parameters, wherein the frequency domain Gaussian mask parameters include frequency, mean, and standard deviation;

[0021] Constructing high-frequency mask and low-frequency mask through the predicted frequency-domain Gaussian mask parameters;

[0022] The primary fusion features are subjected to mask separation operations to obtain high-frequency features and low-frequency features with semantic information.

[0023] Further, the parameter prediction network is:

[0024] F p =f line (f reshape (f pool (f 3 (Relu(f 3 (F)))))), where f 3 (·) is a 3×3 convolutional layer, Relu(·) is a parameterized nonlinear activation function, f pool (·) is the average pooling operation, which pools the spatial dimension to 1, f line (·) is a linear mapping that maps the pooled result to the parameters of the Gaussian mask.

[0025] Further, the construction of the high-frequency mask and the low-frequency mask by using the predicted frequency-domain Gaussian mask parameters includes mask value design, specifically:

[0026] Construct a standard mask grid based on the size of the input primary fusion features;

[0027] In the frequency range obtained by the parameter prediction network, the mask values ​​less than the frequency are set to 0, and the rest are set to 1;

[0028] Gaussian smoothing is performed on the mask values ​​set to 0 to obtain high-frequency features.

[0029] Furthermore, the high-frequency features and low-frequency features are respectively input into the adaptive frequency feature enhancement module to obtain enhanced high-frequency features and low-frequency features. Specifically, the high-frequency features and low-frequency features are inverse Fourier transformed and then the frequency features of each local block are calculated to perform specific frequency enhancement.

[0030] Furthermore, the adaptive frequency feature enhancement module is specifically:

[0031] The high-frequency and low-frequency features obtained by the dynamic high- and low-frequency separation module are inversely Fourier transformed back to the real domain;

[0032] Divide the real domain into windows;

[0033] Input the divided window into MLP to extract the corresponding Gabor frequency parameters;

[0034] The extracted Gabor frequency parameters are input into the two-dimensional Gabor expression to construct the Gabor filter, extract the frequency direction and magnitude, and further obtain the enhanced frequency characteristics.

[0035] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0036] 1. The present invention designs a dynamic frequency enhancement network, which can realize dynamic frequency feature enhancement based on different target images. The structure can extract and enhance the features of different frequencies dynamically, fully solving the problem of excessive frequency difference of input images.

[0037] 2. The present invention designs a dynamic frequency separation module, which can separate the most appropriate high- and low-frequency features according to different feature maps without losing semantic information. The module performs dynamic high- and low-frequency separation by predicting the corresponding separation frequency and Gaussian mask parameters of the input feature map, thereby enhancing the generalization ability.

[0038] 3. The present invention designs an adaptive frequency feature enhancement module, which can adaptively enhance the frequency domain features, calculate the frequency features to be enhanced through a dynamic windowing algorithm, adaptively enhance specific frequencies, and further enhance the sharpening results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a workflow diagram of a hyperspectral image sharpening method based on dynamic frequency enhancement of the present invention;

[0040] Figure 2 It is a schematic diagram of the structure of the primary fusion network of the present invention;

[0041] Figure 3 It is a structural schematic diagram of the dynamic high and low frequency separation module of the present invention;

[0042] Figure 4 It is a structural schematic diagram of the adaptive frequency characteristic enhancement module of the present invention;

[0043] Figure 5 is a GT image used in an embodiment of the present invention;

[0044] Figure 6 is based on Figure 5 Synthetic full-color image;

[0045] Figure 7 It is a fusion image using PNN network;

[0046] Figure 8 It is a fusion image using the DRPNN network;

[0047] Figure 9 It is the fused image using DARN network;

[0048] Figure 10 is the fused image using this method. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto.

[0050] Example

[0051] In this embodiment, a hyperspectral image PaviaU acquired by a ROSIS satellite sensor in the center of Pavia, Italy is used, which includes 103 bands, and the size of the hyperspectral image is 610*340*103.

[0052] like Figures 1-4 As shown, this embodiment provides a hyperspectral image sharpening method based on dynamic frequency enhancement, comprising:

[0053] Step 1: read the hyperspectral image dataset, obtain the panchromatic image by averaging the bands of the hyperspectral image, and then obtain the low-resolution hyperspectral image by downsampling the hyperspectral image, where the spatial resolution ratio of the panchromatic image P to the low-resolution hyperspectral image H is 4, and then divide it into training set, validation set, and test set.

[0054] Specifically:

[0055] For the hyperspectral image H hr ∈R h×w×c The included visible light bands are averaged to obtain the full-color image P∈R h×w×1 , and then downsample the hyperspectral image to obtain the downsampled hyperspectral image The downsampled hyperspectral image is then low-pass filtered and Gaussian blurred to simulate the sampling blur of real-world hyperspectral images to reduce image quality.

[0056] The downsampled hyperspectral image H lr With the hyperspectral image H hr The resolution relationship satisfied is h = h 1 ×r,w=w 1 ×r, where r is the difference in resolution between the hyperspectral image and the panchromatic image.

[0057] In the divided data set, the panchromatic image and the hyperspectral image are sampled correspondingly at a fixed sampling interval to obtain the corresponding data blocks, and then the data images are flipped vertically and horizontally for data enhancement.

[0058] Step 2 constructs a dynamic frequency enhanced full-color sharpening network, wherein the full-color sharpening network includes a primary fusion network, a dynamic high-low frequency separation module, and an adaptive frequency feature enhancement module;

[0059] Specifically:

[0060] The upsampled hyperspectral image H is obtained by interpolation algorithm. lr1 ∈R h×w×c , the upsampled hyperspectral image H lr1 And the full-color image P is spliced ​​to obtain the feature F∈R h×w×c+1 , input feature F into the primary fusion network to obtain the primary fusion feature Input the primary fusion features into the dynamic high and low frequency separation module to obtain high frequency features and low frequency characteristics Then the high-frequency features and low-frequency features are respectively connected to the adaptive frequency feature enhancement module to obtain the enhanced high-frequency features and low frequency characteristics The high-frequency features and low-frequency features are concatenated to obtain Input to a convolution layer for convolutional dimension reduction to get F 7 ∈R h×w×c , and then compared with the original upsampled hyperspectral image H lr1 ∈R h×w×c Perform the addition operation to get the final output H out ∈R h×w×c .

[0061] Furthermore, the primary fusion network includes a convolution layer and n residual convolution layers, the convolution layer has a kernel size of 3×3, a step size of 1, a padding of 0, a padding length of 1, and a mapping dimension of c+1 to c 1 .

[0062] In the n residual convolutional layers, n is set to 3, where each residual layer is as follows: out =f 3 (Relu(f 3 (F)))+F, where F is the forward feature, Relu(·) is the parameterized nonlinear activation function, and f 3 (·) is a 3×3 convolutional layer.

[0063] The concatenated features F∈R h×w×c+1 , first perform a 3×3 convolutional layer to map the input features to Then F k Connect n residual convolution layers, and then add the results of n residual convolution layers to F k The primary fusion features are obtained.

[0064] The dynamic high and low frequency separation module is specifically as follows: input primary fusion feature The predicted frequency domain Gaussian mask parameters are obtained through the parameter prediction network. The frequency domain Gaussian mask parameters include frequency Fre, mean ε and standard deviation σ. The high-frequency mask and low-frequency mask are constructed through the predicted frequency domain Gaussian mask parameters. Then, the primary fusion feature F after Fourier transformation is respectively 1 Perform mask operations to obtain high-frequency features with semantic information and low frequency characteristics

[0065] like Figure 3 As shown, the parameter prediction network is F p =f line (f reshape (f pool (f 3 (Relu(f 3 (F)))))), where f 3 (·) is a 3×3 convolutional layer, Relu(·) is a parameterized nonlinear activation function, f pool (·) is the average pooling operation, which pools the spatial dimension to 1, f line (·) is a linear mapping, which maps the pooled result to a Gaussian mask parameter. After obtaining the Gaussian mask parameter, the Gaussian formula is used Design the mask value, where x, y are the grid coordinates of the Gaussian mask, σ x ,σ y ,μ x ,μ y is the value predicted by the prediction network.

[0066] The specific process of obtaining high-frequency features and low-frequency features is further described as follows:

[0067] The primary fusion feature is Fourier transformed, that is, F f1 =FFT(F 1 ), FFT(·) is Fourier transform.

[0068] The primary fusion features are passed through the parameter prediction network to obtain the frequency, mean and standard deviation. The frequency value determines which frequency features are retained, and the mean standard deviation is used to construct a Gaussian function to attenuate frequency features outside the retained frequency.

[0069] The frequency, mean and standard value are input into the Gaussian formula for Gaussian mask design. The mask value design is as follows:

[0070] Construct a standard mask grid of the same size as the input primary fusion features;

[0071] In order to extract high-frequency features, the corresponding mask value is set to 0 within the frequency range obtained by the parameter prediction network, and the rest is set to 1;

[0072] In order to maintain the integrity of the semantic features, Gaussian smoothing is performed on the mask values ​​that were originally 0. The parameters of Gaussian smoothing are designed based on the prediction parameters of the parameter prediction network. The maximum value of the Gaussian function is 1, and as the distance between the frequency coordinate and the cutoff frequency increases, the value of the Gaussian function gradually decreases;

[0073] After the mask value is scaled by the Gaussian function, the Gaussian mask is finally multiplied with the Fourier transformed input primary fusion feature (mask separation) to obtain the high-frequency feature, i.e., F high =F f1 ·F mask , For the mask.

[0074] The process of acquiring low-frequency features is similar to that of high-frequency features. The standard mask grid and Gaussian smoothing method are also used to effectively extract low-frequency information. This processing method ensures that the extraction of frequency features is both accurate and retains the necessary semantic information.

[0075] Further, if Figure 4 As shown, the adaptive frequency feature enhancement module calculates the frequency features of each local block after inverse Fourier transform of the high-frequency features and the low-frequency features, and performs specific frequency enhancement. The specific implementation process is as follows:

[0076] The high-frequency features and low-frequency features separated by the mask are inversely Fourier transformed back to the real domain, F g1 =IFFT(F highorlow );

[0077] F g1 Divide the window. where h 2 =n 1 ×h 1 , w 2 =n 2 × 1 , w 1 and h 1 To divide the width and height of the window, n 1 and n 2 is the number of cut windows, each window is called a local block;

[0078] By adding F g1_window Input into MLP to extract the corresponding Gabor frequency parameters, where the two-dimensional Gabor expression is The parameter predicted by MLP is σ x ,σy ,ω x ,ω y . The MLP network is F MLP =f line (f reshape (f pool (Relu(f 3 (F g1_window ))))), where F MLP The parameters for prediction

[0079] Input the prediction parameters into the two-dimensional Gabor expression to construct the Gabor filter and extract the features corresponding to the frequency direction and size. g1_window_fre =Gabor*F g1_window , Gabor is the designed filter, * is the convolution operation, and the Gabor filter kernel size is set to 7.

[0080] F g1_window_fre Reverse back to the size of the original feature map, splicing the local enhancement results of the real part and the imaginary part;

[0081] Convolutional fusion F g2_out =f 3 (f cat (F g1_out_real ,F g1_out_im )), where f 3 (·) is a 3×3 convolutional layer, f cat (·) is the splicing operation, F g1_out_real The real part of the inverted feature map after Gabor feature extraction F g1_out_im The imaginary part of the inverted feature map after two-dimensional Gabor feature extraction Finally, the enhanced features are obtained.

[0082] like Figure 4 As shown in the figure, by dividing the window, the input features are divided into many independent local feature blocks (window division), and then each window is input into the MLP to map the Gabor parameters of each local block, and then input into the Gabor function expression to construct a Gabor filter to dynamically enhance the frequency characteristics of each local fast. Since the parameters of each local fast Gabor filter are calculated instead of manually filled, it is an enhancement of a specific frequency.

[0083] The high-frequency features are enhanced by two-dimensional Gabor features F 4 The feature F enhanced by two-dimensional Gabor with low-frequency features5 Fuse and reduce the dimension to the number of original hyperspectral channels, and inject the missing detailed information into the original hyperspectral image to obtain the sharpened hyperspectral image. 7 = f 1 (f 3 (f cat (F 4 , F 5 ))), where f 3 (·) is a 3×3 convolutional layer, f 1 (·) is a 1×1 dimensionality reduction convolutional layer, f cat (·) is a concatenation operation.

[0084] Step 3: Train the pan-sharpening network with dynamic frequency enhancement;

[0085] Step 4: Input the hyperspectral image into the trained pan-sharpening network to obtain the sharpened hyperspectral image.

[0086] The sharpening algorithms compared in this method are: bicubic interpolation method, deep learning-based algorithms PNN, DRPNN, DARN. The dataset compared is PaviaU, which contains 103 bands, the size of the hyperspectral image is 610*340*103, and the panchromatic image is synthesized by averaging the visible light bands of the hyperspectral image. Figure 5 is the GT image, Figure 6 is the synthesized panchromatic image, Figure 7 is the fused image of the PNN network, Figure 8 is the fused image of the DRPNN network, Figure 9 is the fused image of the DARN network, Figure 10 is the fused image of the network proposed in this paper. It can be seen from the displayed results that the fusion effect of this method has good image edges and the best effect.

[0087] In summary, the dynamic frequency enhancement network proposed by the present invention can well enhance the frequency characteristics in the frequency domain, and has strong generalization ability, and can dynamically adjust the network parameters according to different feature maps to achieve dynamic enhancement.

[0088] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement methods and are all included in the protection scope of the present invention.

Claims

1. A hyperspectral image sharpening method based on dynamic frequency enhancement, characterized in that: include: Read the hyperspectral image data set, and process the hyperspectral images in the data set to obtain full-color images and low-resolution hyperspectral images; Constructing a dynamic frequency enhanced full-color sharpening network, the full-color sharpening network includes a primary fusion network, a dynamic high-low frequency separation module and an adaptive frequency feature enhancement module; Training a pan-sharpening network for dynamic frequency enhancement; The hyperspectral image is input into the trained panchromatic sharpening network to obtain the sharpened hyperspectral image.

2. The hyperspectral image sharpening method according to claim 1, characterized in that: The processing of the hyperspectral images in the data set to obtain the panchromatic image and the low-resolution hyperspectral image is specifically: averaging the visible light bands contained in the spectral image to obtain the panchromatic image; The hyperspectral image is Gaussian filtered and then downsampled by s times to obtain a low-resolution hyperspectral image, which is then divided into training set, validation set and test set by insufficient slicing.

3. The hyperspectral image sharpening method according to claim 2, characterized in that: The resolution relationship between the low-resolution hyperspectral image and the hyperspectral image is h=h1×r, w=w1×r, where r is the resolution multiple of the hyperspectral image and the panchromatic image.

4. The hyperspectral image sharpening method according to claim 1, characterized in that: The pan-sharpening network includes a primary fusion network, a dynamic high-low frequency separation module and an adaptive frequency feature enhancement module, and its processing flow is as follows: An upsampled image of the hyperspectral image is obtained, and the upsampled image and the panchromatic image are spliced ​​and input into the primary fusion network to obtain the primary fusion features; Input the primary fusion features into the dynamic high-low frequency separation module to obtain high-frequency features and low-frequency features; The high-frequency features and low-frequency features are respectively input into the adaptive frequency feature enhancement module to obtain the enhanced high-frequency features and low-frequency features. The enhanced high-frequency features and low-frequency features are concatenated and then convolved for dimensionality reduction. They are then added to the upsampled image to obtain a sharpened hyperspectral image.

5. The hyperspectral image sharpening method according to claim 4, characterized in that: The primary fusion network includes a convolutional layer and n residual convolutional layers.

6. The hyperspectral image sharpening method according to claim 1, characterized in that: The processing process of the dynamic high and low frequency separation module is as follows: The primary fusion features are passed through a parameter prediction network to obtain predicted frequency domain Gaussian mask parameters, wherein the frequency domain Gaussian mask parameters include frequency, mean, and standard deviation; Constructing high-frequency mask and low-frequency mask through the predicted frequency-domain Gaussian mask parameters; The primary fusion features are subjected to mask separation operations to obtain high-frequency features and low-frequency features with semantic information.

7. The hyperspectral image sharpening method according to claim 6, characterized in that: The parameter prediction network is: F p =f line (f reshape (f pool (f3(Relu(f3(F)))))), where f3(·) is a 3×3 convolutional layer, Relu(·) is a parameterized nonlinear activation function, and f ppol (·) is the average pooling operation, which pools the spatial dimension to 1, f line (·) is a linear mapping that maps the pooled result to the parameters of the Gaussian mask.

8. The hyperspectral image sharpening method according to claim 6, characterized in that: It also includes mask value design, specifically: Construct a standard mask grid based on the size of the input primary fusion features; In the frequency range obtained by the parameter prediction network, the mask values ​​less than the frequency are set to 0, and the rest are set to 1; Gaussian smoothing is performed on the mask values ​​set to 0 to obtain high-frequency features.

9. The hyperspectral image sharpening method according to claim 4, characterized in that: The high-frequency features and low-frequency features are respectively input into the adaptive frequency feature enhancement module to obtain enhanced high-frequency features and low-frequency features. Specifically, the high-frequency features and low-frequency features are inverse Fourier transformed and then the frequency features of each local block are calculated to perform frequency enhancement.

10. The hyperspectral image sharpening method according to claim 1, characterized in that: The adaptive frequency feature enhancement module is specifically: The high-frequency and low-frequency features obtained by the dynamic high- and low-frequency separation module are inversely Fourier transformed back to the real domain; Divide the real domain into windows; Input the divided window into MLP to extract the corresponding Gabor frequency parameters; The extracted Gabor frequency parameters are input into the two-dimensional Gabor expression to construct the Gabor filter, extract the frequency direction and magnitude, and further obtain the enhanced frequency characteristics.

Images