Convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance

Abstract

The invention provides a hyperspectral remote sensing image sharpening method based on a convolutional neural network guided by spectral features, and the method comprises the steps: obtaining a hyperspectral image and a panchromatic image in the same region, and carrying out the preprocessing; constructing a training data set and a test data set according to the preprocessed hyperspectral image and panchromatic image; constructing a sharpening convolutional neural network model comprising a spectral feature extraction module, a feature fusion module and a spectral guide module; training the sharpened convolutional neural network model based on the training data set to obtain a target neural network model; and inputting the hyperspectral image and the panchromatic image in the test data set into the target neural network model to obtain a high-spatial-resolution hyperspectral image. According to the invention, the spectral fidelity of high-spatial-resolution hyperspectral remote sensing image sharpening is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance. Background Art

[0002] Remote sensing image technology is widely used in many fields such as resource exploration, earth research and military. At the same time, with the continuous development of technology in these fields, higher requirements are placed on the spatial resolution and spectral resolution of remote sensing images. However, due to the physical limitations of sensors, it is usually difficult to directly obtain hyperspectral images with high spatial resolution. In order to overcome this limitation, the usual practice is to simultaneously obtain two types of image data: one is a hyperspectral image with high spectral resolution but low spatial resolution, and the other is a panchromatic image with high spatial resolution but a single band, and then the two are fused through sharpening technology to generate remote sensing images with high spatial resolution and high spectral resolution. Traditional hyperspectral image panchromatic sharpening methods can be divided into three categories, including component substitution method, multi-resolution analysis method and model optimization-based method. Component substitution method mainly includes a series of sharpening methods based on domain transformation technology such as principal component analysis and Schmidt orthogonal transform method; multi-resolution analysis method is represented by wavelet transform method and its variants, Laplace pyramid transform method and brightness smoothing filter adjustment algorithm; and model optimization-based methods mainly include sparse representation-based methods and Bayesian algorithms.

[0003] With the remarkable progress made in sharpening by convolutional neural networks, hyperspectral image sharpening methods based on convolutional neural networks have shown better sharpening effects than traditional methods. These methods use the powerful feature extraction capabilities of convolutional neural networks to simultaneously process hyperspectral images and panchromatic images to improve image quality. However, when the spatial resolution of panchromatic and hyperspectral images is relatively large, the spectral distortion of the sharpened results is very serious. Moreover, many existing sharpening convolutional networks start with the overall sharpening effect, and by stacking deeper convolutional layers, the network spontaneously pays attention to the spatial-spectral relationship of hyperspectral images; however, the lack of clear guidance, coupled with the small amount of data in remote sensing image datasets, makes it difficult for the convolutional network to fully learn the spectral mapping relationship behind it, resulting in poor spectral fidelity of existing sharpening results. Summary of the invention

[0004] The present invention provides a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance, which improves the spectral fidelity of high spatial resolution hyperspectral remote sensing image sharpening.

[0005] In a first aspect, the present invention provides a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance, comprising:

[0006] Acquire hyperspectral images and panchromatic images of the same area and perform preprocessing;

[0007] Construct training data set and test data set based on the preprocessed hyperspectral image and panchromatic image;

[0008] Construct a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module and a spectral guidance module;

[0009] The sharpened convolutional neural network model is trained based on the training data set to obtain a target neural network model; the hyperspectral image and the panchromatic image in the test data set are input into the target neural network model to obtain a high spatial resolution hyperspectral image.

[0010] Optionally, the preprocessing includes:

[0011] Performing low-pass filtering on the hyperspectral image and the panchromatic image respectively to obtain a first hyperspectral image and a first panchromatic image; wherein the panchromatic image and the hyperspectral image have different spatial resolutions;

[0012] Downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image; wherein the second panchromatic image and the first hyperspectral image have the same spatial resolution;

[0013] The second hyperspectral image is upsampled by using a polynomial interpolation method to obtain a third hyperspectral image; wherein the third hyperspectral image has the same spatial resolution as the first hyperspectral image.

[0014] Optionally, the sharpening convolutional neural network model further includes: a spectral buffer convolution layer, a spectral reconstruction convolution layer and a spectral compensation layer;

[0015] The training of the sharpened convolutional neural network model based on the training data set includes:

[0016] Inputting the hyperspectral image in the training data set into the spectral feature extraction module to output a spectral feature map; and inputting the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image into the feature fusion module to output a fusion feature map;

[0017] Inputting the spectral feature map and the fused feature map into the spectral guidance module, and outputting a first feature map;

[0018] Inputting the first feature map into the spectral buffer convolution layer, and outputting a second feature map;

[0019] Inputting the second feature map into the spectrum reconstruction convolution layer, and outputting a third feature map;

[0020] The hyperspectral image in the training data set and the third feature map are input into the spectral compensation layer, and a predicted image is output.

[0021] Optionally, the spectral buffer convolution layer includes 64 convolution kernels with a receptive field of 1×1; the spectral reconstruction convolution layer includes 64 convolution kernels with a receptive field of 1×1; and the spectral compensation layer is used to perform pixel-by-pixel addition processing on the hyperspectral image in the training data set and the third feature map.

[0022] Optionally, the spectral feature extraction module includes three first three-dimensional convolutional layers connected in series, three second three-dimensional convolutional layers connected in series, three third three-dimensional convolutional layers connected in series and an aggregated convolutional layer; wherein the convolution kernel size of the first three-dimensional convolutional layer is smaller than the convolution kernel size of the second three-dimensional convolutional layer, and the convolution kernel size of the second three-dimensional convolutional layer is smaller than the convolution kernel size of the third three-dimensional convolutional layer.

[0023] Optionally, the feature fusion module includes: a first feature fusion convolution layer with a convolution kernel of 3×3, a second feature fusion convolution layer with a convolution kernel of 5×5, and a third feature fusion convolution layer with a convolution kernel of 7×7;

[0024] The feature fusion module is used to input the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image, and output a fused feature map, including:

[0025] Input the hyperspectral image into a first feature fusion convolution layer, and output a first fusion feature map;

[0026] Input the first fused feature map and the full-color image into two second feature fused convolutional layers connected in series, and output a second fused feature map;

[0027] Input the second fused feature map and the full-color image into two third feature fusion convolutional layers connected in series, and output a third fused feature map;

[0028] The third fused feature map and the full-color image are input into the first feature fusion convolution layer, and a fused feature map is output.

[0029] Optionally, the spectral guidance module includes: a spectral guidance convolution layer, a spectral fusion convolution layer, a spectral guidance convolution layer, and a spectral fusion convolution layer;

[0030] The spectral guidance module is used to input the spectral feature map into the spectral guidance convolution layer and output a first guidance feature map; input the first guidance feature map and the fused feature map output by the feature fusion module into the spectral fusion convolution layer and then pass through the spectral guidance convolution layer to obtain a second guidance feature map; input the second guidance feature map and the fused feature map into the spectral fusion convolution layer and output a first feature map.

[0031] In a second aspect, the present invention provides a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance, comprising:

[0032] A preprocessing module is used to obtain hyperspectral images and panchromatic images of the same area and perform preprocessing;

[0033] A construction module is used to construct a training data set and a test data set based on the preprocessed hyperspectral image and panchromatic image; a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module and a spectral guidance module;

[0034] A training module is used to train the sharpened convolutional neural network model based on the training data set to obtain a target neural network model; the hyperspectral image and the panchromatic image in the test data set are input into the target neural network model to obtain a high spatial resolution hyperspectral image.

[0035] In a third aspect, an embodiment of the present invention further provides a computing device, comprising a memory and a target processor, wherein the memory stores a computer program, and when the target processor executes the computer program, the method described in any first aspect of this specification is implemented.

[0036] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method described in any first aspect of this specification.

[0037] The present invention provides a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance. The method first obtains a hyperspectral image of the same area and a panchromatic image aligned therewith, and performs preprocessing. Then, a training data set and a test data set are constructed based on the preprocessed hyperspectral image and panchromatic image. A sharpening convolutional neural network including a spectral feature extraction module, a feature fusion module and a spectral guidance module is designed. After training, a target neural network model is obtained. At this time, the test data set is input into the target neural network model to obtain a high spatial resolution hyperspectral image. In this way, the present invention designs a spectral feature guided sharpening convolutional neural network, which can effectively extract the spectral features of the output hyperspectral image, so as to guide the fusion of the hyperspectral image and the panchromatic image according to the vectorial nature of the spectral features, improve the spectral fidelity of the sharpening result, and effectively improve the overall sharpening performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance provided by an embodiment of the present invention;

[0040] Figure 2 It is a structural diagram of a sharpening convolutional neural network model provided by one embodiment of the present invention;

[0041] Figure 3 is a structural diagram of a spectral feature extraction module provided by an embodiment of the present invention;

[0042] Figure 4 is a structural diagram of a feature fusion module provided by an embodiment of the present invention;

[0043] Figure 5 is a structural diagram of a spectrum guiding module provided by an embodiment of the present invention;

[0044] FIG6(a) is a PaviaU hyperspectral reference image; FIG6(b) is a hyperspectral image processed by bicubic interpolation; FIG6(c) is an image processed by a principal component analysis algorithm based on guided filtering; FIG6(d) is an image processed by a brightness smoothing filter adjustment algorithm; FIG6(e) is an image processed by a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance according to an embodiment of the present invention;

[0045] Figure 7 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

[0046] Figure 8 It is a structural schematic diagram of a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, an embodiment of the present invention provides a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance, the method comprising:

[0049] Step 100, obtaining a hyperspectral image and a panchromatic image of the same area, and performing preprocessing;

[0050] Step 102, constructing a training data set and a test data set based on the preprocessed hyperspectral image and panchromatic image;

[0051] Step 104, constructing a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module and a spectral guidance module;

[0052] Step 106, training the sharpened convolutional neural network model based on the training data set to obtain a target neural network model; inputting the hyperspectral image and the panchromatic image in the test data set into the target neural network model to obtain a high spatial resolution hyperspectral image.

[0053] In the present invention, a hyperspectral image of the same area and a panchromatic image aligned therewith are first obtained and preprocessed, and then a training data set and a test data set are constructed based on the preprocessed hyperspectral image and panchromatic image, and a sharpening convolutional neural network including a spectral feature extraction module, a feature fusion module and a spectral guidance module is designed. After training, a target neural network model is obtained, and the test data set is input into the target neural network model to obtain a high spatial resolution hyperspectral image. In this way, the present invention designs a spectral feature guided sharpening convolutional neural network, which can effectively extract the spectral features of the output hyperspectral image, so as to guide the fusion of the hyperspectral image and the panchromatic image according to the vector nature of the spectral features, improve the spectral fidelity of the sharpening result, and effectively improve the overall sharpening performance.

[0054] With respect to step 100, the preprocessing includes:

[0055] Performing low-pass filtering on the hyperspectral image and the panchromatic image respectively to obtain a first hyperspectral image and a first panchromatic image; wherein the panchromatic image and the hyperspectral image have different spatial resolutions;

[0056] Downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image; wherein the second panchromatic image and the first hyperspectral image have the same spatial resolution;

[0057] The second hyperspectral image is upsampled by using a polynomial interpolation method to obtain a third hyperspectral image; wherein the third hyperspectral image has the same spatial resolution as the first hyperspectral image.

[0058] Specifically, for example, hyperspectral images and full color images The spatial dimensions of have the following quantitative relationship:

[0059]

[0060] Among them, h 1 and w 1 Respectively represent the length and width of the hyperspectral image; h 2 and w 2 They represent the length and width of the panchromatic image, respectively; r represents the ratio of the spatial resolution of the panchromatic image to the hyperspectral image; b represents the number of channels of the hyperspectral image;

[0061] Hyperspectral images and full color images A frequency response adaptive smoothing filter is used to perform low-pass filtering to obtain a first hyperspectral image and a first panchromatic image;

[0062] The first hyperspectral image and the first panchromatic image are downsampled to reduce their spatial resolution by r times to obtain the second hyperspectral image. and the second full-color image Wherein, the second panchromatic image and the first hyperspectral image have the same spatial resolution;

[0063] The second hyperspectral image is upsampled using polynomial interpolation to increase its spatial resolution by r times, and a third hyperspectral image with the same spatial resolution as the first hyperspectral image is obtained.

[0064] In step 102, a training data set and a test data set are constructed, including: and the second full-color image Processing, screening a specific area, and then cutting out a small-sized sub-image without overlap in the area according to a fixed sampling interval and These sub-images are used as training samples; the order of these samples is randomly shuffled to ensure the randomness of the training data, thus constructing a training data set for training the model. Next, the third hyperspectral image is selected. The remaining area and the full-color image corresponding to the corresponding position Cut it into test samples and construct a test dataset.

[0065] It should be noted that both the training data set and the test data set include the preprocessed hyperspectral image (i.e., the third hyperspectral image) and panchromatic image (i.e., the corresponding second panchromatic image) of the same area as input, and the high spatial resolution hyperspectral image of the corresponding area as output.

[0066] For step 104, if Figure 2 As shown, the sharpening convolutional neural network model also includes: a spectral buffer convolution layer, a spectral reconstruction convolution layer and a spectral compensation layer;

[0067] The training of the sharpened convolutional neural network model based on the training data set includes:

[0068] Inputting the hyperspectral image in the training data set into the spectral feature extraction module to output a spectral feature map; and inputting the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image into the feature fusion module to output a fusion feature map;

[0069] Inputting the spectral feature map and the fused feature map into the spectral guidance module, and outputting a first feature map;

[0070] Inputting the first feature map into the spectral buffer convolution layer, and outputting a second feature map;

[0071] Inputting the second feature map into the spectrum reconstruction convolution layer, and outputting a third feature map;

[0072] The hyperspectral image in the training data set and the third feature map are input into the spectral compensation layer, and a predicted image is output.

[0073] In the present invention, the spectral feature extraction module is used to extract spectral features that are consistent with the input hyperspectral image spectrum, wherein the loss function is used for constraint. The feature fusion module is used to perform preliminary feature fusion on the input hyperspectral image and panchromatic image, and complete preliminary sharpening. The spectral guidance module is used to guide the fused features using the spectral feature map to ensure the consistency of its spectrum with the original input hyperspectral image, and perform the second step of sharpening.

[0074] In a preferred embodiment, Figure 3 As shown, the spectral feature extraction module includes three first three-dimensional convolutional layers connected in series, three second three-dimensional convolutional layers connected in series, three third three-dimensional convolutional layers connected in series, and an aggregated convolutional layer; wherein the convolution kernel size of the first three-dimensional convolutional layer is smaller than the convolution kernel size of the second three-dimensional convolutional layer, and the convolution kernel size of the second three-dimensional convolutional layer is smaller than the convolution kernel size of the third three-dimensional convolutional layer.

[0075] The spectral feature extraction module designed in the present invention has three branches, each of which is equipped with three-dimensional convolutions of different sizes, which can effectively extract spectral information of different resolutions of hyperspectral images, and use the loss function of calculating the spectral angle to constrain the extracted spectral features to ensure that its spectrum is highly consistent with the original hyperspectral image.

[0076] Specifically, for example, the structure of the spectral feature extraction module is as follows: Figure 3 As shown, including:

[0077] The three-dimensional convolution layer Conv1_1 contains 64 convolution kernels with a receptive field of 3×3×3 and a convolution step size of s=1. The hyperspectral image in the input training dataset Output 64-channel feature map The operation process can be expressed as Where W 1 and B 1 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0078] The three-dimensional convolution layer Conv1_2 contains 64 convolution kernels with a receptive field of 3×3×3 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 2 and B 2 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0079] The three-dimensional convolution layer Conv1_3 contains 64 convolution kernels with a receptive field of 3×3×3 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 3 and B 3 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0080] The three-dimensional convolution layer Conv2_1 contains 64 convolution kernels with a receptive field of 5×5×5 and a convolution step size of s=1. Input feature map Output 64-channel feature map The operation process can be expressed as Where W 4 and B 4 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear LeakyRelu activation function;

[0081] The three-dimensional convolution layer Conv2_2 contains 64 convolution kernels with a receptive field of 5×5×5 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 5 and B 5 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0082] The three-dimensional convolution layer Conv2_3 contains 64 convolution kernels with a receptive field of 5×5×5 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 6 and B 6 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0083] The three-dimensional convolution layer Conv3_1 contains 64 convolution kernels with a receptive field of 7×7×7 and a convolution step size of s=1. Input feature map Output 64-channel feature map The operation process can be expressed as Where W 7 and B 7Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear LeakyRelu activation function;

[0084] The three-dimensional convolution layer Conv3_2 contains 64 convolution kernels with a receptive field of 7×7×7 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 8 and B 8 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0085] The three-dimensional convolution layer Conv3_3 contains 64 convolution kernels with a receptive field of 7×7×7 and a convolution step size of s=1. Input feature map Output 64 channel feature maps The operation process can be expressed as Where W 9 and B 9 Respectively represent the weight and bias of the convolution kernel of this layer, Represents the nonlinear Leaky Relu activation function;

[0086] Convolutional layer Conv4, which contains 64 convolution kernels with a receptive field of 5×5×5. Input feature map and Output 64-channel spectral feature map The operation process can be expressed as in and Respectively represent the weight and bias of the convolution kernel of this layer, Indicates concatenating multiple feature maps in the channel dimension, φ l (·) also represents the nonlinear LeakyRelu activation function.

[0087] In a preferred embodiment, Figure 4 As shown, the feature fusion module includes: a first feature fusion convolution layer with a convolution kernel of 3×3, a second feature fusion convolution layer with a convolution kernel of 5×5, and a third feature fusion convolution layer with a convolution kernel of 7×7;

[0088] The feature fusion module is used to input the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image, and output a fused feature map, including:

[0089] Input the hyperspectral image into a first feature fusion convolution layer, and output a first fusion feature map;

[0090] Input the first fused feature map and the full-color image into two second feature fused convolutional layers connected in series, and output a second fused feature map;

[0091] Input the second fused feature map and the full-color image into two third feature fusion convolutional layers connected in series, and output a third fused feature map;

[0092] The third fused feature map and the full-color image are input into the first feature fusion convolution layer, and a fused feature map is output.

[0093] In the present invention, the feature fusion module inputs a low spatial resolution hyperspectral image and a corresponding panchromatic image, and the feature fusion convolution extracts and fuses the spatial spectral information. In addition, the panchromatic image has multiple branches, which are input into convolution layers of different depths to compensate for the possible loss of spatial information during the fusion process; finally, the fully fused spatial spectral features (i.e., the fused feature map) are output.

[0094] Specifically, the structure of the feature fusion module is as follows: Figure 4 As shown, including:

[0095] Feature fusion convolution layer Conv1, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input hyperspectral image in the training dataset Output 64-channel feature map The operation process can be expressed as Where W 1 and B 1 Respectively represent the weight and bias of the convolution kernel of this layer, φ l (x) represents the nonlinear Relu activation function;

[0096] Feature fusion convolution layer Conv2, which contains 64 convolution kernels with a receptive field of 5×5 and a convolution step size of s=1. Input feature map and the corresponding hyperspectral images in the training dataset Full color image Output 64-channel feature map The operation process can be expressed as Where W 2 and B 2 Respectively represent the weight and bias of the convolution kernel of this layer, Indicates concatenating multiple feature maps in the channel dimension, φ l (x) represents the nonlinear Relu activation function;

[0097] Feature fusion convolution layer Conv3, which contains 64 convolution kernels with a receptive field of 5×5 and a convolution step size of s=1. Input feature map Output 64-channel feature map The operation process can be expressed as Where W 3 and B 3 Respectively represent the weight and bias of the convolution kernel of this layer, φ l (x) represents the nonlinear Relu activation function;

[0098] Feature fusion convolution layer Conv4, which contains 64 convolution kernels with a receptive field of 7×7 and a convolution step size of s=1. Input feature map and the corresponding full-color image in the training set Output 64-channel feature map The operation process can be expressed as Where W 4 and B 4 Respectively represent the weight and bias of the convolution kernel of this layer, Indicates concatenating multiple feature maps in the channel dimension, φ l (x) represents the nonlinear Relu activation function;

[0099] Feature fusion convolution layer Conv5, which contains 64 convolution kernels with a receptive field of 7×7 and a convolution step size of s=1. Input feature map Output 64-channel feature map The operation process can be expressed as Where W 5 and B 5 Respectively represent the weight and bias of the convolution kernel of this layer, φ l (x) represents the nonlinear Relu activation function;

[0100] Feature fusion convolution layer Conv6, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input feature map and the corresponding full-color image in the training set Output 64-channel fusion feature map The operation process can be expressed as Where W 6 and B 6 Respectively represent the weight and bias of the convolution kernel of this layer, Indicates concatenating multiple feature maps in the channel dimension, φ l (x) represents the nonlinear ReLU activation function.

[0101] In a preferred embodiment, Figure 5 As shown, the spectral guidance module includes: a spectral guidance convolution layer, a spectral fusion convolution layer, a spectral guidance convolution layer, and a spectral fusion convolution layer;

[0102] The spectral guidance module is used to input the spectral feature map into the spectral guidance convolution layer and output a first guidance feature map; input the first guidance feature map and the fused feature map output by the feature fusion module into the spectral fusion convolution layer and then pass through the spectral guidance convolution layer to obtain a second guidance feature map; input the second guidance feature map and the fused feature map into the spectral fusion convolution layer and output a first feature map.

[0103] The spectral guidance module designed in the present invention first multiplies the spectral features obtained by the spectral feature extraction module and the spatial spectrum features obtained by the feature fusion module pixel by pixel, corrects the spatial spectrum features through spectral pixels, and the subsequent fusion convolution further sharpens the corrected spatial spectrum features to improve the sharpening effect.

[0104] Specifically, the structure of the spectral guidance module is as follows: Figure 5 As shown, including:

[0105] Spectral guided convolution layer Conv1, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input is the spectral feature map output by the spectral feature extraction module Output 64-channel feature map The operation process can be expressed as Where W 1 and B 1 Respectively represent the weight and bias of the convolution kernel of this layer, φ l (x) represents the nonlinear Relu activation function;

[0106] Spectral fusion convolution layer Conv2, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input feature map And the fused feature map output by the feature fusion module Output 64-channel feature map The operation process can be expressed as Where W 2 and B 2 Respectively represent the weight and bias of the convolution kernel of this layer, Indicates that pixel-by-pixel multiplication, φ l (x) represents the nonlinear Relu activation function;

[0107] Spectral guided convolution layer Conv3, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input feature map Output 64-channel feature map The operation process can be expressed as Where W 3 and B 3 Respectively represent the weight and bias of the convolution kernel of this layer, φl (x) represents the nonlinear Relu activation function;

[0108] Spectral fusion convolution layer Conv4, which contains 64 convolution kernels with a receptive field of 3×3 and a convolution step size of s=1. Input feature map And the fused feature map output by the feature fusion module Output 64-channel first feature map The operation process can be expressed as Where W 4 and B 4 Respectively represent the weight and bias of the convolution kernel of this layer, ⊕ means adding pixel by pixel, φ l (x) represents the nonlinear ReLU activation function.

[0109] In a preferred embodiment, the spectral buffer convolution layer includes 64 convolution kernels with a receptive field of 1×1; the spectral reconstruction convolution layer includes 64 convolution kernels with a receptive field of 1×1; and the spectral compensation layer is used to perform pixel-by-pixel addition processing on the hyperspectral image in the training data set and the third feature map.

[0110] Continuing from the previous example, Figure 2 As shown in Figure 1, the spectral buffer convolution layer Conv1 contains 64 convolution kernels with a receptive field of 1×1. Input the first feature map Output 64 second feature maps The operation process can be expressed as Where W 1 and B 1 They represent the weight and bias of the convolution kernel of this layer respectively, and φ(·) represents the nonlinear ReLU activation function;

[0111] The spectral reconstruction convolution layer Conv2 contains 64 convolution kernels with a receptive field of 1×1. Input the second feature map Output b third feature maps The operation process can be expressed as Where W 2 and B 2 They represent the weight and bias of the convolution kernel of this layer respectively. This layer does not use activation function;

[0112] Spectral compensation layer Compense. Input the hyperspectral image in the training dataset And the third characteristic map Output prediction image O (i) The operation process can be expressed as Where ⊕ represents a pixel-by-pixel addition operation.

[0113] For step 106, select appropriate hyperparameters and initialize the network parameters of the sharpened convolutional neural network model. Based on the training data set, select appropriate loss function and optimization algorithm to train the sharpened convolutional neural network model. After the loss function value reaches the minimum, save the optimal model at this time to obtain the target neural network model.

[0114] Specifically, the loss function selected in step 106 is a composite loss function, and its expression is:

[0115] L(θ)=L 1 (θ)+λL 2 (θ)

[0116] Where L 1 (θ) is the mean absolute error loss function, which is expressed as:

[0117]

[0118] Where Ω represents the input-output mapping relationship of the sharpened convolutional neural network model; θ represents the parameters of the sharpened convolutional neural network; N b Indicates the number of training samples input in each round of iteration; ||·|| F represents the Frobenius norm; is the hyperspectral image in the training dataset; is the panchromatic image corresponding to the hyperspectral image in the training dataset; Corresponding to the training data set and The reference image (i.e., the high spatial resolution hyperspectral image as output in the training set); where L 2 (θ) is the cosine loss function, which is expressed as:

[0119]

[0120] Among them, i is the pixel number; α i ∈(-π,π] represents the spectral characteristic graph Hyperspectral images in the training set The spectral angle between them is as follows: 2 (θ) is smaller; λ is the penalty factor; h 2 and w 2 They are respectively the length and width of the full-color image in the training set.

[0121] The high spatial resolution hyperspectral image sharpening method based on spectral feature guidance provided by the present invention is based on the above-mentioned spectral feature extraction module, feature fusion module and spectral guidance module, and constructs a spectral feature guided sharpening convolutional neural network. The network constructs a structure for spectral processing, extracts the spectral features of the input image, and guides the fusion of subsequent spatial spectral features according to the vectorial nature of the spectral features, ensuring that the spectral characteristics of the spatial spectral features are highly consistent with the input image, thereby improving the spectral fidelity of the sharpening result.

[0122] Specifically, the present invention uses the PaviaU hyperspectral image and panchromatic image from the ROSIS imaging spectrometer to verify the sharpening effect. The hyperspectral image has 103 spectral channels and a spatial resolution of 32×32; the panchromatic image has a spatial resolution of 160×160, and the ratio of the spatial resolutions of the two is 1:5.

[0123] Specifically, FIG6(a) is a PaviaU hyperspectral reference image, FIG6(b) is a hyperspectral image obtained after bicubic interpolation, FIG6(c) is an image processed using a conventional principal component analysis algorithm based on guided filtering, FIG6(d) is an image processed using a conventional brightness smoothing filter adjustment algorithm, and FIG6(e) is an image processed using the sharpening method of this embodiment. As shown in FIG6, the image processed using the bicubic interpolation method loses a large amount of spatial detail information compared to the reference image, and exhibits severe spatial distortion; the image processed using the principal component analysis algorithm based on guided filtering achieves a certain degree of sharpening, but still has certain spatial blur and spectral distortion; the image processed using the brightness smoothing filter adjustment algorithm achieves a better global sharpening effect and has a higher spectral fidelity, but still has the problem of spatial detail loss in some areas; and the image processed using the method described in the embodiment of the present invention achieves the best sharpening effect, presents better spatial details and has an outstanding spectral fidelity effect, and has the best overall effect.

[0124] like Figure 7 , Figure 8 As shown, the embodiment of the present invention provides a hyperspectral remote sensing image sharpening device based on a convolutional neural network guided by spectral features. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Figure 7 As shown, it is a hardware architecture diagram of a computing device where a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance provided by an embodiment of the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown in the figure, the computing device in which the device is located in the embodiment may also generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 8As shown, as a device in a logical sense, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance, and the device includes:

[0125] A preprocessing module 800 is used for the preprocessing module to obtain a hyperspectral image and a panchromatic image of the same area and perform preprocessing;

[0126] A construction module 802 is used to construct a training data set and a test data set according to the preprocessed hyperspectral image and panchromatic image; and to construct a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module, and a spectral guidance module;

[0127] The training module 804 is used to train the sharpened convolutional neural network model based on the training data set to obtain a target neural network model; the hyperspectral image and the panchromatic image in the test data set are input into the target neural network model to obtain a high spatial resolution hyperspectral image.

[0128] In some specific implementations, the preprocessing module 800 may be used to execute the above step 100 , the construction module 802 may be used to execute the above step 102 and step 104 , and the training module 804 may be used to execute the above step 106 .

[0129] In one embodiment of the present invention, the preprocessing module 800 is further configured to perform the following operations:

[0130] Performing low-pass filtering on the hyperspectral image and the panchromatic image respectively to obtain a first hyperspectral image and a first panchromatic image; wherein the panchromatic image and the hyperspectral image have different spatial resolutions;

[0131] Downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image; wherein the second panchromatic image and the first hyperspectral image have the same spatial resolution;

[0132] The second hyperspectral image is upsampled by using a polynomial interpolation method to obtain a third hyperspectral image; wherein the third hyperspectral image has the same spatial resolution as the first hyperspectral image.

[0133] In one embodiment of the present invention, the sharpening convolutional neural network model further includes: a spectral buffer convolution layer, a spectral reconstruction convolution layer and a spectral compensation layer;

[0134] The training module 804 is also used to perform the following operations:

[0135] Inputting the hyperspectral image in the training data set into the spectral feature extraction module to output a spectral feature map; and inputting the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image into the feature fusion module to output a fusion feature map;

[0136] Inputting the spectral feature map and the fused feature map into the spectral guidance module, and outputting a first feature map;

[0137] Inputting the first feature map into the spectral buffer convolution layer, and outputting a second feature map;

[0138] Inputting the second feature map into the spectrum reconstruction convolution layer, and outputting a third feature map;

[0139] The hyperspectral image in the training data set and the third feature map are input into the spectral compensation layer, and a predicted image is output.

[0140] In one embodiment of the present invention, the spectral buffer convolution layer includes 64 convolution kernels with a receptive field of 1×1; the spectral reconstruction convolution layer includes 64 convolution kernels with a receptive field of 1×1; and the spectral compensation layer is used to perform pixel-by-pixel addition processing on the hyperspectral image in the training data set and the third feature map.

[0141] In one embodiment of the present invention, the spectral feature extraction module includes three first three-dimensional convolutional layers connected in series, three second three-dimensional convolutional layers connected in series, three third three-dimensional convolutional layers connected in series and an aggregated convolutional layer; wherein the convolution kernel size of the first three-dimensional convolutional layer is smaller than the convolution kernel size of the second three-dimensional convolutional layer, and the convolution kernel size of the second three-dimensional convolutional layer is smaller than the convolution kernel size of the third three-dimensional convolutional layer.

[0142] In one embodiment of the present invention, the feature fusion module includes: a first feature fusion convolution layer with a convolution kernel of 3×3, a second feature fusion convolution layer with a convolution kernel of 5×5, and a third feature fusion convolution layer with a convolution kernel of 7×7;

[0143] The feature fusion module is used to input the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image, and output a fused feature map, including:

[0144] Input the hyperspectral image into a first feature fusion convolution layer, and output a first fusion feature map;

[0145] Input the first fused feature map and the full-color image into two second feature fused convolutional layers connected in series, and output a second fused feature map;

[0146] Input the second fused feature map and the full-color image into two third feature fusion convolutional layers connected in series, and output a third fused feature map;

[0147] The third fused feature map and the full-color image are input into the first feature fusion convolution layer, and a fused feature map is output.

[0148] In one embodiment of the present invention, the spectral guidance module includes: a spectral guidance convolution layer, a spectral fusion convolution layer, a spectral guidance convolution layer, and a spectral fusion convolution layer;

[0149] The spectral guidance module is used to input the spectral feature map into the spectral guidance convolution layer and output a first guidance feature map; input the first guidance feature map and the fused feature map output by the feature fusion module into the spectral fusion convolution layer and then pass through the spectral guidance convolution layer to obtain a second guidance feature map; input the second guidance feature map and the fused feature map into the spectral fusion convolution layer and output a first feature map.

[0150] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance. In other embodiments of the present invention, a convolutional neural network hyperspectral remote sensing image sharpening device based on spectral feature guidance may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0151] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0152] An embodiment of the present invention also provides a computing device, including a memory and a target processor, wherein a computer program is stored in the memory, and when the target processor executes the computer program, a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance in any embodiment of the present invention is implemented.

[0153] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a convolutional neural network hyperspectral remote sensing image sharpening method based on spectral feature guidance in any embodiment of the present invention.

[0154] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0155] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0156] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0157] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling the operating system on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0158] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0159] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.

[0160] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral remote sensing image sharpening method based on convolutional neural network guided by spectral features, characterized in that: include: Acquire hyperspectral images and panchromatic images of the same area and perform preprocessing; Construct training data set and test data set based on the preprocessed hyperspectral image and panchromatic image; Construct a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module and a spectral guidance module; The sharpened convolutional neural network model is trained based on the training data set to obtain a target neural network model; the hyperspectral image and the panchromatic image in the test data set are input into the target neural network model to obtain a high spatial resolution hyperspectral image.

2. The method according to claim 1, characterized in that The pre-processing comprises: Performing low-pass filtering on the hyperspectral image and the panchromatic image respectively to obtain a first hyperspectral image and a first panchromatic image; wherein the panchromatic image and the hyperspectral image have different spatial resolutions; Downsampling the first hyperspectral image and the first panchromatic image to obtain a second hyperspectral image and a second panchromatic image; wherein the second panchromatic image and the first hyperspectral image have the same spatial resolution; The second hyperspectral image is upsampled by using a polynomial interpolation method to obtain a third hyperspectral image; wherein the third hyperspectral image has the same spatial resolution as the first hyperspectral image.

3. The method according to claim 1, characterized in that The sharpening convolutional neural network model also includes: a spectral buffer convolution layer, a spectral reconstruction convolution layer and a spectral compensation layer; The training of the sharpened convolutional neural network model based on the training data set includes: Inputting the hyperspectral image in the training data set into the spectral feature extraction module to output a spectral feature map; and inputting the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image into the feature fusion module to output a fusion feature map; Inputting the spectral feature map and the fused feature map into the spectral guidance module, and outputting a first feature map; Inputting the first feature map into the spectral buffer convolution layer, and outputting a second feature map; Inputting the second feature map into the spectrum reconstruction convolution layer, and outputting a third feature map; The hyperspectral image in the training data set and the third feature map are input into the spectral compensation layer, and a predicted image is output.

4. The method according to claim 3, characterized in that The spectral buffer convolution layer includes 64 convolution kernels with a receptive field of 1×1; the spectral reconstruction convolution layer includes 64 convolution kernels with a receptive field of 1×1; the spectral compensation layer is used to perform pixel-by-pixel addition processing on the hyperspectral image in the training data set and the third feature map.

5. The method according to claim 1, characterized in that The spectral feature extraction module includes three first three-dimensional convolutional layers connected in series, three second three-dimensional convolutional layers connected in series, three third three-dimensional convolutional layers connected in series and an aggregated convolutional layer; wherein the convolution kernel size of the first three-dimensional convolutional layer is smaller than the convolution kernel size of the second three-dimensional convolutional layer, and the convolution kernel size of the second three-dimensional convolutional layer is smaller than the convolution kernel size of the third three-dimensional convolutional layer.

6. The method according to claim 1, characterized in that The feature fusion module includes: a first feature fusion convolution layer with a convolution kernel of 3×3, a second feature fusion convolution layer with a convolution kernel of 5×5, and a third feature fusion convolution layer with a convolution kernel of 7×7; The feature fusion module is used to input the hyperspectral image in the training data set and the panchromatic image corresponding to the hyperspectral image, and output a fused feature map, including: Input the hyperspectral image into a first feature fusion convolution layer, and output a first fusion feature map; Input the first fused feature map and the full-color image into two second feature fused convolutional layers connected in series, and output a second fused feature map; Input the second fused feature map and the full-color image into two third feature fusion convolutional layers connected in series, and output a third fused feature map; The third fused feature map and the full-color image are input into the first feature fusion convolution layer, and a fused feature map is output.

7. The method according to any one of claims 1 to 6, characterized in that: The spectral guidance module includes: a spectral guidance convolution layer, a spectral fusion convolution layer, a spectral guidance convolution layer, and a spectral fusion convolution layer; The spectral guidance module is used to input the spectral feature map into the spectral guidance convolution layer and output a first guidance feature map; input the first guidance feature map and the fused feature map output by the feature fusion module into the spectral fusion convolution layer and then pass through the spectral guidance convolution layer to obtain a second guidance feature map; input the second guidance feature map and the fused feature map into the spectral fusion convolution layer and output a first feature map.

8. A hyperspectral remote sensing image sharpening device based on a convolutional neural network guided by spectral features, characterized in that: include: A preprocessing module is used to obtain hyperspectral images and panchromatic images of the same area and perform preprocessing; A construction module is used to construct a training data set and a test data set based on the preprocessed hyperspectral image and panchromatic image; a sharpening convolutional neural network model including a spectral feature extraction module, a feature fusion module and a spectral guidance module; A training module is used to train the sharpened convolutional neural network model based on the training data set to obtain a target neural network model; the hyperspectral image and the panchromatic image in the test data set are input into the target neural network model to obtain a high spatial resolution hyperspectral image.

9. A computing device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images