Double-branch hyperspectral spatial super-resolution network model product, training method and application

Abstract

The invention discloses a double-branch hyperspectral spatial super-resolution network model product, a training method and application, and belongs to the field of image processing. The double-branch hyperspectral spatial super-resolution network model comprises a demixing branch network, a non-demixing branch network and a fusion network; the unmixing branch network comprises an unmixing module, an abundance image super-resolution module and a linear mixing module, and is used for processing and outputting an unmixing super-resolution hyperspectral image; the non-unmixing branch network comprises a spectrum dimension reduction module and a hyperspectral image super-resolution module and is used for processing and outputting a non-unmixing super-resolution hyperspectral image; and the fusion network performs spectrum dimension raising on the non-unmixing super-resolution hyperspectral image, and performs cascading and grouping convolution dimension reduction corresponding to each wave band on the image after dimension raising and the unmixing super-resolution hyperspectral image to obtain a high-resolution hyperspectral image. And the combination effect of the spatial-spectral information is improved, so that the spatial super-resolution quality of the hyperspectral image is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and more specifically, relates to a dual-branch high-spectral spatial super-resolution network model product, a training method and an application. Background Art

[0002] Hyperspectral images have the unique advantage of "image-spectrum integration", which can provide the intrinsic structure and material properties of substances, and help improve the accuracy of tasks such as material detection and analysis. However, due to the limitations of hardware conditions, the spatial resolution of hyperspectral images is usually low, which limits the development of hyperspectral images. Therefore, the research on spatial super-resolution of hyperspectral images is of great significance.

[0003] Hyperspectral image spatial super-resolution methods are mainly divided into single-image-based and fusion-based hyperspectral image spatial super-resolution reconstruction methods. The fusion-based hyperspectral image super-resolution reconstruction method can effectively improve the quality of hyperspectral image spatial super-resolution by using multispectral images with high spatial resolution but low spectral resolution in the same scene as auxiliary information. However, this method usually requires strict alignment of the contents of multispectral images and hyperspectral images. In contrast, the single-image-based hyperspectral image spatial super-resolution method explores directly improving the spatial resolution of low-resolution hyperspectral images without using any auxiliary information, and therefore has higher practical application value.

[0004] Hyperspectral image spatial super-resolution methods based on single images lack the assistance of multispectral images with high spatial resolution and are more challenging. Traditional single-image super-resolution methods include sparse regularization methods, maximum a posteriori methods, and low-rank decomposition methods, which use prior knowledge to restore the details of low-resolution images. However, traditional methods find it difficult to obtain rich image information of hyperspectral images, resulting in limited effects of spatial super-resolution reconstruction.

[0005] The single-image super-resolution method based on deep learning has achieved better super-resolution reconstruction effect than the traditional method. The hyperspectral image spatial super-resolution method based on deep learning can be divided into hyperspectral image spatial super-resolution based on unmixing and hyperspectral image spatial super-resolution without unmixing. The hyperspectral image super-resolution method based on unmixing obtains the end members and corresponding abundance map of the hyperspectral image by unmixing, and obtains a high-resolution hyperspectral image by linearly mixing the spatial super-resolution of the abundance image with the unmixed end members. This method is conducive to the spectral continuity after super-resolution, but the number of end members needs to be set by human experience, which limits the effect of super-resolution. Hyperspectral image spatial super-resolution based on non-unmixing focuses on the utilization of spectral information, while the traditional hyperspectral super-resolution network focuses on the reconstruction quality of the spatial aspect and ignores the continuity of spectral information. Although the current hyperspectral spatial super-resolution method has made progress and the quality of spatial super-resolution has been improved, the problem of insufficient combination of spatial and spectral information still exists, and the accuracy of spatial super-resolution still needs to be improved. Summary of the invention

[0006] In view of the defects of the related art, the purpose of the present invention is to provide a dual-branch hyperspectral spatial super-resolution network model product, training method and application, aiming to solve the problems that spatial spectral information is difficult to combine in hyperspectral image super-resolution and the spatial super-resolution accuracy is not high.

[0007] To achieve the above-mentioned object, in a first aspect, the present invention provides a dual-branch hyperspectral spatial super-resolution network model product, including a dual-branch hyperspectral spatial super-resolution network model;

[0008] The dual-branch hyperspectral spatial super-resolution network model includes: an unmixing branch network, a non-unmixing branch network and a fusion network;

[0009] The unmixing branch network includes an unmixing module, an abundance map super-resolution module and a linear mixing module; the unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members; the abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map; the linear mixing module is used to linearly mix the high-resolution abundance map and the end members to obtain an unmixed super-resolution hyperspectral image;

[0010] The non-unmixed branch network includes a spectral dimension reduction module and a hyperspectral image super-resolution module; the spectral dimension reduction module is used to perform spectral dimension reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimension-reduced low-resolution hyperspectral image to obtain a non-unmixed super-resolution hyperspectral image;

[0011] The fusion network is used to perform spectral dimension upscaling on the non-unmixed super-resolution hyperspectral image, and to perform cascading and group convolution dimension reduction on the dimension upscaling non-unmixed super-resolution hyperspectral image and the unmixed super-resolution hyperspectral image corresponding to each band, so as to obtain a high-resolution hyperspectral image.

[0012] Optionally, the abundance map super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule;

[0013] The encoding submodule is used to downsample the input data twice to obtain an encoding feature map; the decoding submodule is used to upsample the encoding feature map twice to obtain a decoding feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoding feature map and the decoding feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

[0014] Optionally, the unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members, including:

[0015] The unmixing module unmixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding end members, and preliminarily performs convolution scoring on the low-resolution abundance map to obtain a weight score for each abundance map, and then selects the low-resolution abundance maps corresponding to the top K maximum scores through the learned adaptive threshold to obtain the selected low-resolution abundance map and the corresponding end members.

[0016] Optionally, the hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule;

[0017] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a non-demixed super-resolution hyperspectral image.

[0018] In a second aspect, the present invention further provides a method for training a dual-branch high-spectral spatial super-resolution network model, wherein a dual-branch high-spectral spatial super-resolution network model of a dual-branch high-spectral spatial super-resolution network model product as described in any one of the first aspects is trained to obtain a trained dual-branch high-spectral spatial super-resolution network model;

[0019] The training loss includes spectral information loss and spatial information loss;

[0020] The spatial information loss includes the spatial L1 loss of the super-resolution image and the spatial L1 loss of the unmixing reconstruction; the expression of the spatial information loss is:

[0021]

[0022] Among them, Y1 is the non-unmixed super-resolution hyperspectral image after dimensionality increase, Y2 is the unmixed super-resolution hyperspectral image, and Y3 is the high-resolution hyperspectral image. is the real high-resolution hyperspectral image, X is the input low-resolution hyperspectral image, X r It is the reconstructed low-resolution hyperspectral image after the low-resolution abundance map and the linear mixing of the end members;

[0023] The spectral information loss includes the spectral information loss L of the super-resolution image sam The spectral information loss L of the unmixed reconstruction sam ; The expression of the spectral information loss is:

[0024]

[0025] In a third aspect, the present invention further provides a hyperspectral image spatial super-resolution method, which is applied to the trained dual-branch hyperspectral spatial super-resolution network model as described in the second aspect, comprising:

[0026] Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

[0027] In a fourth aspect, the present invention further provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the hyperspectral image spatial super-resolution method as described in the third aspect.

[0028] Compared with the prior art, the above technical solution conceived by the present invention can achieve the following beneficial effects:

[0029] 1. The present invention provides a dual-branch hyperspectral spatial super-resolution network model. The super-resolution result obtained by the spatial super-resolution of the abundance map of the unmixing branch and the linear mixing of the end members is helpful for the utilization of spectral information, thereby improving the smoothness between spectral bands after the hyperspectral spatial super-resolution reconstruction; the spatial super-resolution of the original hyperspectral image of the non-unmixing branch helps to improve the super-resolution reconstruction effect in the image space; the fusion network performs spectral dimension upscaling on the non-unmixed super-resolution hyperspectral image, and performs cascading and group convolution dimension reduction on the non-unmixed super-resolution hyperspectral image and the unmixed super-resolution hyperspectral image after dimension upscaling corresponding to each band, to obtain a high-resolution hyperspectral image. Therefore, compared with the existing hyperspectral super-resolution technology, it is more conducive to the combination of spatial and spectral information, thereby improving the quality of hyperspectral image spatial super-resolution.

[0030] 2. A dual-branch hyperspectral spatial super-resolution network model provided by the present invention has the same structure as the abundance map super-resolution module in the unmixed branch network and the hyperspectral image super-resolution module in the non-unmixed branch network, and both improve the final super-resolution effect through multi-scale spatial super-resolution images and fusion. Existing technologies often use structures such as Transformer to improve the utilization of global spatial information, but the model is more complex and less efficient. The multi-scale spatial super-resolution method improves the network's receptive field through spatial dimensionality reduction and convolution, and helps to extract richer super-resolution features through multi-scale spatial super-resolution, thereby effectively improving the final super-resolution effect.

[0031] 3. A hyperspectral image spatial super-resolution method provided by the present invention is applied to a dual-branch hyperspectral spatial super-resolution network model provided by the present invention, and a low-resolution hyperspectral image is input into the dual-branch hyperspectral spatial super-resolution network model, and a high-resolution hyperspectral image is output after a series of image processing. Since the abundance map super-resolution module and the hyperspectral image super-resolution module have the same structure, the overall model complexity is low, and the efficiency of super-resolution reconstruction is improved; and since the combination of spatial-spectral information is fully utilized, the quality of the output high-resolution hyperspectral image is improved.

[0032] 4. A Top-k selection unmixing method provided by the present invention is applied to the unmixing branch network provided by the present invention, and the preliminary low-resolution abundance map and the corresponding end members obtained by unmixing are convoluted and scored preliminarily to obtain the weight score of each abundance map, and then the low-resolution abundance maps corresponding to the top K maximum scores are selected through the learned adaptive threshold to obtain the selected low-resolution abundance map and the corresponding end members. The existing technology limits the accuracy of unmixing by artificially setting a fixed number of abundance maps and end members, resulting in poor super-resolution effect of hyperspectral images based on unmixing. The Top-k selection unmixing method can adaptively select an appropriate number of abundance maps and end members through network training, thereby effectively improving the effect of spatial super-resolution of hyperspectral images. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a framework of a dual-branch hyperspectral spatial super-resolution network model provided by an example of the present invention;

[0034] Figure 2 A schematic diagram of a demixing branch network provided by an example of the present invention;

[0035] Figure 3 A network block diagram of the abundance map super-resolution module provided in the example of the present invention for multi-scale spatial super-resolution reconstruction;

[0036] Figure 4 A schematic diagram of a non-demixing branch network provided by an example of the present invention;

[0037] Figure 5 A network block diagram of a hyperspectral image super-resolution module provided by an example of the present invention for multi-scale spatial super-resolution reconstruction;

[0038] Figure 6 A schematic diagram of a fusion network after dual-branch super-resolution provided for an example of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0040] The contents involved in the above embodiment are described below in conjunction with a preferred embodiment.

[0041] Embodiment 1

[0042] like Figure 1As shown, the present invention provides a dual-branch high-spectral spatial super-resolution network model product, including a dual-branch high-spectral spatial super-resolution network model;

[0043] The dual-branch hyperspectral spatial super-resolution network model includes: an unmixing branch network, a non-unmixing branch network and a fusion network;

[0044] The unmixing branch network includes an unmixing module, an abundance map super-resolution module and a linear mixing module; the unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members; the abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map; the linear mixing module is used to linearly mix the high-resolution abundance map and the end members to obtain an unmixed super-resolution hyperspectral image;

[0045] The non-unmixed branch network includes a spectral dimension reduction module and a hyperspectral image super-resolution module; the spectral dimension reduction module is used to perform spectral dimension reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimension-reduced low-resolution hyperspectral image to obtain a non-unmixed super-resolution hyperspectral image;

[0046] The fusion network is used to perform spectral dimension upscaling on the non-unmixed super-resolution hyperspectral image, and to perform cascading and group convolution dimension reduction on the dimension upscaling non-unmixed super-resolution hyperspectral image and the unmixed super-resolution hyperspectral image corresponding to each band, so as to obtain a high-resolution hyperspectral image.

[0047] Among them, Figure 1 As shown in Figure 1, the dual-branch network shares the same input, which is a low-resolution hyperspectral image, and the output of each branch is a reconstructed high-resolution image. Among them, the spatial resolution is h×w and the band length is L.

[0048] like Figure 2 As shown in Figure 2, for the hyperspectral image spatial super-resolution network branch based on unmixing, the input is first unmixed to obtain a low-resolution abundance map. and the corresponding end member E, where c is the number of end members. The unmixing module uses an adaptive Top-k selection module to obtain a low-resolution abundance map after k selections. and the corresponding end member E′; after multi-scale spatial super-resolution of the abundance map super-resolution module, k high-resolution abundance maps are obtained; the high-resolution abundance map is linearly mixed with the end members of the unmixing network to obtain the final unmixed super-resolution hyperspectral image.

[0049] like Figure 4As shown in the figure, for the hyperspectral image spatial super-resolution branch network based on non-unmixing, the spectral dimension reduction module is a three-layer group convolution; wherein, the convolution kernel size is 3×3, the number of groups in the first layer of convolution is L, and the output channel is L; the second layer of convolution starts spectral dimension reduction, the number of groups in the convolution is L / / 2, and the output channel is L / / 2; the number of groups in the third layer of convolution is L / / 4, and the output channel is L / / 4; wherein, L is the number of input spectral bands, each layer of convolution has Batch norm and Relu function activation, and the super-resolution network is a multi-scale spatial super-resolution network.

[0050] Further, such as Figure 6 As shown in the figure, the fusion network first performs spectral dimension upscaling on the output of the non-unmixed super-resolution network (non-unmixed super-resolution hyperspectral image) to L, and then performs cascade corresponding to each band with the result of the unmixed super-resolution network (unmixed super-resolution hyperspectral image), downsamples to L through grouped convolution, and finally obtains the final output through a layer of grouped convolution; wherein, spectral dimension upscaling adopts bilinear interpolation, the first grouped convolution input channel is 2L, the number of groups is L, the output channel is L, and the convolution kernel size is 3×3; the last layer of convolution output channel is L, the number of groups is L, and the convolution kernel size is 3×3, wherein each layer of convolution has Batch norm and Relu function activation.

[0051] In the embodiment of the present invention, a dual-branch hyperspectral spatial super-resolution network model is constructed, including a hyperspectral image spatial super-resolution network branch based on unmixing and a hyperspectral image spatial super-resolution network branch based on non-unmixing. The super-resolution result obtained by the spatial super-resolution of the abundance map of the unmixing branch and the linear mixing of the end members is helpful for the utilization of spectral information, thereby improving the smoothness between spectral bands after hyperspectral spatial super-resolution reconstruction. The original hyperspectral image spatial super-resolution of the non-unmixing branch is helpful for improving the super-resolution reconstruction effect in the image space. The fusion network cascades the super-resolution hyperspectral images obtained by the two branch networks for each band to obtain the final high-resolution hyperspectral image. The technical problems of the difficulty in combining spatial spectral information in hyperspectral image super-resolution and the low precision of high spatial super-resolution are solved. The utilization rate and combination effect of spatial spectral information are improved, thereby improving the quality of hyperspectral image spatial super-resolution.

[0052] Based on the above embodiment, optionally, the unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members, including:

[0053] The unmixing module unmixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding end members, and preliminarily performs convolution scoring on the low-resolution abundance map to obtain a weight score for each abundance map, and then selects the low-resolution abundance maps corresponding to the top K maximum scores through the learned adaptive threshold to obtain the selected low-resolution abundance map and the corresponding end members.

[0054] like Figure 2 and Figure 3 As shown in the figure, the number of convolution layers of the unmixing module is designed to be 4, the convolution kernel size is 3×3, the number of channels is 8c, 4c, 2c and c, respectively, and c is the number of end members set; each convolution layer has the activation of Batch norm and Relu function, and the activation function of the last layer is Softmax to meet the constraint of sum to 1. The spectrum of the input low-resolution hyperspectral image is projected to 8c, 4c, 2c and c through 4 convolution layers in turn, and c preliminary low-resolution abundance maps a and corresponding end members E are obtained. The low-resolution abundance map a can meet the prior constraints of non-negative and sum to 1.

[0055] The c low-resolution abundance maps a are spatially downsampled twice with a convolution kernel of 3×3 and a stride of 2. Each downsampled abundance map is averaged and Softmax activated to obtain the score of each low-resolution abundance map. The scores are selected through an adaptive threshold to obtain the low-resolution abundance maps a′ and the corresponding end members E′ corresponding to the top k maximum scores, thereby completing the Top-k selection.

[0056] The end member E′ is designed as L 1×1 convolution kernels, where L is the number of bands of the hyperspectral image. The low-resolution hyperspectral reconstruction image X is obtained by convolving a′. r , so it satisfies:

[0057] X r =a′E′

[0058] The difference between the reconstructed image and the input is used as the loss, and the weight of the unmixing module can be obtained through training.

[0059] The low-resolution abundance map obtained after unmixing is subjected to multi-scale spatial super-resolution to obtain Among them, the spatial resolution after super resolution is H × W. Therefore, the final unmixed super-resolution hyperspectral image Y u satisfy:

[0060] Y u =AE′

[0061] Furthermore, for the super-resolution network branch based on non-demixed hyperspectral images, the input X is first spectrally downsampled to obtain Where l is the spectral dimension after downsampling.

[0062] The downsampling method is group convolution to maintain the order of spectral bands. Group convolution is used as a downsampling operation. The downsampling ratio is determined by the number of groups and the output channel. The number of spectra after each convolution is reduced by half. The hyperspectral image after spectral downsampling is then obtained through multi-scale spatial super-resolution of the hyperspectral image. The final output is obtained by upsampling the grouped convolution Among them, upsampling is also performed through grouped convolution.

[0063] Furthermore, for the fusion network, the output Y of the two branches is u and Y non-u The corresponding bands are cascaded in pairs to maintain the order of the bands. The image is downsampled to L through group convolution and then passed through another layer of group convolution to better fuse the super-resolution image of the two branches. Therefore, the final upsampled image after fusion can be expressed as:

[0064] Y=DWConv(Cat((Y non-u ,Y u ))

[0065] Among them, DWConv(·) represents group convolution and Cat(·) represents cascade operation.

[0066] Optionally, the abundance map super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule;

[0067] The encoding submodule is used to downsample the input data twice to obtain an encoding feature map; the decoding submodule is used to upsample the encoding feature map twice to obtain a decoding feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoding feature map and the decoding feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

[0068] The input data is encoded with two downsampling steps and decoded with two upsampling steps to obtain an output of the same size. The feature maps of different sizes in the encoding and decoding parts are upsampled to the size of the super-resolution image, and the final super-resolution image is obtained through cascading and fusion.

[0069] Specifically, the given input is The resolution is h×w, and L is the number of features. Downsampling encoding operation is performed to obtain the encoded features i=[1,2,…,n]. Satisfies:

[0070]

[0071] Among them, n is the number of downsampling layers, Down(·) is the downsampling operation, and maximum pooling is used.

[0072] Each convolution includes two group convolutions to keep the order of spectral bands or abundance maps unchanged. The number of feature channels in the encoding intermediate layer is 4L.

[0073] Perform decoding operation to obtain decoding features j=[1,2,…,n]. Satisfies:

[0074]

[0075] Where n is the number of upsampling layers, Up(·) is the upsampling, and the present invention adopts bilinear interpolation. In addition, the decoded input The last coded feature Right now Each upsampling includes two group convolutions, and the number of channels in the decoding middle layer is 4L.

[0076] Feature Layer i=[0,1,…,n] and j=[1,2,…,n], upsample to the super-resolution image size to obtain super-resolution features and satisfy:

[0077]

[0078]

[0079] For the super-resolution of the last layer of decoding features, cascade input is required :

[0080]

[0081] The super-resolution results of each scale space are cascaded and convolved to obtain the super-resolution output. :

[0082]

[0083] Among them, the group convolution includes two layers. The first convolution keeps the number of channels unchanged, and the second convolution reduces the number of channels to the same as the input channels.

[0084] Specifically, they include:

[0085] like Figure 3As shown in the figure, the first downsampling includes two convolutions, batch norm and Relu function activation layers, and finally a 2×2 maximum pooling layer; the size of the convolution kernel of the convolution layer is 3×3, and the convolution method is grouped convolution; the input channel of the first convolution is L, the number of groups is L, and the output channel is 4L; the input channel of the second convolution is 4L, the number of groups is L, and the output channel is 4L.

[0086] The second downsampling includes two convolutions, batch norm and Relu function activation layers, and finally a 2×2 maximum pooling layer; the size of the convolution kernel of the convolution layer is 3×3, and the convolution method is grouped convolution; the input channels of the two convolutions are 4L, the number of groups is L, and the output channels are 4L.

[0087] The first upsampling includes two convolutions, batch norm and activation layers of the Relu function, and finally a bilinear interpolation with an upsampling scale of 2. The size of the convolution kernel of the convolution layer is 3×3, and the convolution method is grouped convolution. The input channels of the two convolutions are 4L, the number of groups is L, and the output channels are 4L.

[0088] The second upsampling includes two convolutions, batch norm and activation layers of the Relu function, and finally a bilinear interpolation with a sampling scale of 2; the size of the convolution kernel of the convolution layer is 3×3, and the convolution method is grouped convolution; the input of the first convolution is cascaded with the feature map after the first downsampling, the input channel of the convolution is 8L, the number of groups is L, and the output channel is 4L; the input channel of the second convolution is 4L, the number of groups is L, and the output channel is L.

[0089] The feature map of the smallest size is upsampled to the super-resolution size, including one convolution, batch norm and activation layer of the Relu function, and finally connected to the bilinear interpolation with a sampling scale of 4K; the size of the convolution kernel is 3×3, and the convolution method is grouped convolution; the input channel of the convolution is 4L, the number of groups is L, the output channel is L, and K is the super-resolution ratio.

[0090] The feature map of the intermediate size is upsampled to the super-resolution size, including a convolution, batch norm and activation layer of the Relu function, and finally a bilinear interpolation with an upsampling scale of 2K; the size of the convolution kernel is 3×3, and the convolution method is grouped convolution; the input channel of the convolution is 4L, the number of groups is L, the output channel is L, and K is the super-resolution ratio.

[0091] The feature map of the largest size is cascaded and input and then upsampled to the super-resolution size, including one convolution, Batchnorm and Relu activation layers, and finally connected to bilinear interpolation with an upsampling scale of K; the size of the convolution kernel is 3×3, and the convolution method is grouped convolution; the input channel of the convolution is 2L, the number of groups is L / / 4, the output channel is L, and K is the super-resolution ratio.

[0092] The feature maps after super-resolution are cascaded and convolved, including two convolutions, batch norm, and activation layers of the Relu function; the size of the convolution kernel of the first convolution is 3×3, and the convolution method is grouped convolution; the input channel of the convolution is 3L, the number of groups is L, and the output channel is 3L; the size of the convolution kernel of the second convolution is 3×3, and the convolution method is grouped convolution; the input channel of the convolution is 3L, the number of groups is L, and the output channel is L.

[0093] Optionally, the hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule;

[0094] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a non-demixed super-resolution hyperspectral image.

[0095] The hyperspectral image super-resolution module has the same structure and similar functions as the abundance map super-resolution module, and will not be described in detail here. Figure 5 and as described in the Abundance Map Super-Resolution Module above.

[0096] On the basis of the above embodiments, the present invention also provides a training method for a dual-branch high-spectral spatial super-resolution network model, which trains the dual-branch high-spectral spatial super-resolution network model of the dual-branch high-spectral spatial super-resolution network model product described in any one of the above embodiments to obtain a trained dual-branch high-spectral spatial super-resolution network model;

[0097] The training loss includes spectral information loss and spatial information loss;

[0098] The spatial information loss includes the spatial L1 loss of the super-resolution image and the spatial L1 loss of the unmixing reconstruction; the expression of the spatial information loss is:

[0099]

[0100] Among them, Y1 is the non-unmixed super-resolution hyperspectral image after dimensionality increase, Y2 is the unmixed super-resolution hyperspectral image, and Y3 is the high-resolution hyperspectral image. is the real high-resolution hyperspectral image, X is the input low-resolution hyperspectral image, X r The reconstructed low-resolution high-spectral image is the selected low-resolution abundance map and the corresponding end-member linear mixture;

[0101] The spectral information loss includes the spectral information loss L of the super-resolution image sam The spectral information loss L of the unmixed reconstruction sam ; The expression of the spectral information loss is:

[0102]

[0103] The loss function includes spatial information loss and spectral information loss; the spatial information loss includes the reconstruction spatial information loss in the unmixing branch and the super-resolution image spatial information loss in the non-unmixing branch, and the spectral information loss includes the spectral loss in the unmixing branch and the super-resolution image spectral loss in the non-unmixing branch. Therefore, the final loss is: L all =L spectral +L spatial .

[0104] Embodiment 2

[0105] The present invention also provides a hyperspectral image spatial super-resolution method, which uses the trained dual-branch hyperspectral spatial super-resolution network model as described in any one of the embodiments, comprising:

[0106] Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

[0107] The low-resolution hyperspectral image is input into the dual-branch hyperspectral spatial super-resolution network model provided in the embodiment, and after a series of image processing, a high-resolution hyperspectral image is output. Since the abundance map super-resolution module and the hyperspectral image super-resolution module have the same structure, the overall model complexity is low, and the efficiency of super-resolution reconstruction is improved; and since the combination of spatial and spectral information is fully utilized, the quality of the output high-resolution hyperspectral image is improved.

[0108] A hyperspectral image spatial super-resolution method provided by an embodiment of the present invention is implemented based on the trained double-branch hyperspectral spatial super-resolution network model provided by any embodiment of the present invention, and has corresponding beneficial effects.

[0109] Embodiment 3

[0110] The present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the hyperspectral image spatial super-resolution method as described in Example 2.

[0111] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A dual-branch hyperspectral spatial super-resolution network model product, characterized in that: Including a dual-branch hyperspectral spatial super-resolution network model; The dual-branch hyperspectral spatial super-resolution network model includes: an unmixing branch network, a non-unmixing branch network and a fusion network; The unmixing branch network includes an unmixing module, an abundance map super-resolution module and a linear mixing module; the unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members; the abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map; the linear mixing module is used to linearly mix the high-resolution abundance map and the end members to obtain an unmixed super-resolution hyperspectral image; The non-unmixed branch network includes a spectral dimension reduction module and a hyperspectral image super-resolution module; the spectral dimension reduction module is used to perform spectral dimension reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimension-reduced low-resolution hyperspectral image to obtain a non-unmixed super-resolution hyperspectral image; The fusion network is used to perform spectral dimension upscaling on the non-unmixed super-resolution hyperspectral image, and to perform cascading and group convolution dimension reduction on the dimension upscaling non-unmixed super-resolution hyperspectral image and the unmixed super-resolution hyperspectral image corresponding to each band, so as to obtain a high-resolution hyperspectral image.

2. The model product according to claim 1, characterized in that: The abundance map super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule; The encoding submodule is used to downsample the input data twice to obtain an encoding feature map; the decoding submodule is used to upsample the encoding feature map twice to obtain a decoding feature map; The super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoding feature map and the decoding feature map to obtain multiple high-resolution feature maps; The fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

3. The model product according to claim 1, characterized in that: The unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and end members, including: the unmixing module unmixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding end members, and preliminarily convolutionally scores the low-resolution abundance map to obtain a weight score for each abundance map, and then selects the low-resolution abundance maps corresponding to the top K maximum scores through a learned adaptive threshold to obtain the selected low-resolution abundance map and the corresponding end members.

4. The model product according to claim 1, characterized in that: The hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule and a fusion submodule; The encoding submodule is used to downsample the input data twice to obtain an encoding feature map; the decoding submodule is used to upsample the encoding feature map twice to obtain a decoding feature map; The super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoding feature map and the decoding feature map to obtain multiple high-resolution feature maps; The fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a non-unmixed super-resolution hyperspectral image.

5. A training method for a dual-branch hyperspectral spatial super-resolution network model, characterized in that: Training the double-branched high-spectral spatial super-resolution network model of the double-branched high-spectral spatial super-resolution network model product according to any one of claims 1 to 4 to obtain a trained double-branched high-spectral spatial super-resolution network model; The training loss includes spectral information loss and spatial information loss; The spatial information loss includes the spatial L1 loss of the super-resolution image and the spatial L1 loss of the unmixing reconstruction; the expression of the spatial information loss is: Among them, Y1 is the non-unmixed super-resolution hyperspectral image after dimensionality increase, Y2 is the unmixed super-resolution hyperspectral image, and Y3 is the high-resolution hyperspectral image. is the real high-resolution hyperspectral image, X is the input low-resolution hyperspectral image, X r A reconstructed low-resolution hyperspectral image after linear mixing of the selected low-resolution abundance map and the corresponding end members; The spectral information loss includes the spectral information loss L of the super-resolution image sam and the spectral information loss L of the unmixed reconstruction sam ; The expression of the spectral information loss is:

6. A hyperspectral image spatial super-resolution method, characterized in that: The method of applying the trained dual-branch hyperspectral spatial super-resolution network model as claimed in claim 5 comprises: Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

7. A machine-readable storage medium storing machine-executable instructions, wherein when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the hyperspectral image spatial super-resolution method according to claim 6.

Images