A dual-branch hyperspectral spatial super-resolution network model product, training method and application

Abstract

The application discloses a kind of dual-branch hyperspectral spatial super-resolution network model product, training method and application, belong to image processing field.Dual-branch hyperspectral spatial super-resolution network model includes unmixing branch network, non-unmixing branch network and fusion network;Unmixing branch network includes unmixing module, abundance map super-resolution module and linear mixing module, and process output unmixing super-resolution hyperspectral image;Non-unmixing branch network includes spectral dimension reduction module and hyperspectral image super-resolution module, and process output non-unmixing super-resolution hyperspectral image;Fusion network carries out spectral dimensionality to non-unmixing super-resolution hyperspectral image, and after image and unmixing super-resolution hyperspectral image are upgraded, each waveband corresponding cascade and grouping convolution dimension reduction are carried out, and high-resolution hyperspectral image is obtained.The combination effect of air spectrum information is improved, so as to improve the quality of hyperspectral image spatial super-resolution.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and more specifically, relates to a dual-branch hyperspectral spatial super-resolution network model product, training method, and application. Background Technology

[0002] Hyperspectral imaging possesses the unique advantage of "image-spectrum integration," providing insights into the intrinsic structure and material properties of substances, thus enhancing the accuracy of tasks such as substance detection and analysis. However, limited by hardware constraints, the spatial resolution of hyperspectral images is typically low, thereby restricting their development. Therefore, research on spatial super-resolution of hyperspectral images is of great significance.

[0003] Hyperspectral image spatial super-resolution methods are mainly divided into single-image-based and fusion-based methods. Fusion-based methods effectively improve the quality of hyperspectral image spatial super-resolution by using multispectral images with high spatial resolution but low spectral resolution in the same scene as auxiliary information. However, this method usually requires strict alignment of the contents of the multispectral and hyperspectral images. In contrast, single-image-based methods explore ways to directly improve the spatial resolution of low-resolution hyperspectral images without using any auxiliary information, thus having greater practical application value.

[0004] Single-image-based hyperspectral image spatial super-resolution methods face greater challenges due to the lack of high-spatial-resolution multispectral images. Traditional single-image super-resolution methods, including sparse regularization, maximum a posteriori (MAP), and low-rank decomposition (LDR) methods, recover details from low-resolution images using prior knowledge. However, these traditional methods struggle to capture the rich image information of hyperspectral images, resulting in limited spatial super-resolution reconstruction performance.

[0005] Deep learning-based single-image super-resolution methods have achieved better super-resolution reconstruction results than traditional methods. Deep learning-based hyperspectral image spatial super-resolution methods can be divided into unmixing-based and unmixing-free methods. Unmixing-based methods obtain endmembers and corresponding abundance maps of the hyperspectral image through unmixing, and then linearly mix the spatial super-resolution of the abundance map with the unmixed endmembers to obtain a high-resolution hyperspectral image. This method is beneficial for the spectral continuity after super-resolution, but the number of endmembers needs to be set empirically, thus limiting the super-resolution effect. Unmixing-free methods focus on utilizing spectral information; traditional hyperspectral super-resolution networks concentrate on spatial reconstruction quality, neglecting the continuity of spectral information. Although current hyperspectral spatial super-resolution methods have made progress and improved the quality of spatial super-resolution, the problem of insufficient combination of spatial and spectral information still exists, and the accuracy of spatial super-resolution still has room for improvement. Summary of the Invention

[0006] To address the shortcomings of related technologies, the present invention aims to provide a dual-branch hyperspectral spatial super-resolution network model product, training method, and application, which aims to solve the problems of difficulty in combining spatial spectral information and low spatial super-resolution accuracy in hyperspectral image super-resolution.

[0007] To achieve the above objectives, in a first aspect, the present invention provides a dual-branch hyperspectral spatial super-resolution network model product, including a dual-branch hyperspectral spatial super-resolution network model.

[0008] The dual-branch hyperspectral spatial super-resolution network model includes: a demixed branch network, a non-demixed branch network, and a fusion network;

[0009] The unmixing branch network includes an unmixing module, an abundance map super-resolution module, and a linear mixing module. The unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers. The abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map. The linear mixing module is used to linearly mix the high-resolution abundance map and endmembers to obtain an unmixed super-resolution hyperspectral image.

[0010] The unmixed branch network includes a spectral dimensionality reduction module and a hyperspectral image super-resolution module; the spectral dimensionality reduction module is used to perform spectral dimensionality reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimensionality-reduced low-resolution hyperspectral image to obtain an unmixed super-resolution hyperspectral image.

[0011] The fusion network is used to perform spectral upscaling on the undissolved super-resolution hyperspectral image, and then concatenates the upscaled undissolved super-resolution hyperspectral image with the undissolved super-resolution hyperspectral image for each band and performs grouped convolutional dimensionality reduction to obtain a high-resolution hyperspectral image.

[0012] Optionally, the abundance map super-resolution module includes an encoding sub-module, a decoding sub-module, a super-resolution sub-module, and a fusion sub-module;

[0013] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

[0014] Optionally, the demixing module is used to demix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers, including:

[0015] The demixing module demixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding endmembers. It also performs a preliminary convolutional scoring on the low-resolution abundance map to obtain a weight score for each abundance map. Then, it selects the K low-resolution abundance maps corresponding to the highest scores through a learned adaptive threshold to obtain the selected low-resolution abundance map and corresponding endmembers.

[0016] Optionally, the hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule, and a fusion submodule;

[0017] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain an unmixed super-resolution hyperspectral image.

[0018] In a second aspect, the present invention also provides a training method for a bi-branch hyperspectral spatial super-resolution network model, wherein the bi-branch hyperspectral spatial super-resolution network model of the bi-branch hyperspectral spatial super-resolution network model product as described in any one of the first aspects is trained to obtain the trained bi-branch hyperspectral spatial super-resolution network model.

[0019] Training loss includes spectral information loss and spatial information loss;

[0020] The spatial information loss includes the spatial L1 loss of the super-resolution image and the spatial L1 loss of the demixing reconstruction; the expression for the spatial information loss is:

[0021]

[0022] Wherein, Y1 is the upscaled, unmixed super-resolution hyperspectral image, Y2 is the unmixed super-resolution hyperspectral image, and Y3 is the high-resolution hyperspectral image. For a true high-resolution hyperspectral image, X is the input low-resolution hyperspectral image, X r The reconstructed low-resolution hyperspectral image is obtained by combining the low-resolution abundance map with the linear mixing of endmembers.

[0023] The spectral information loss includes the spectral information loss L of the super-resolution image. sam The spectral information loss L during unmixing reconstruction sam The expression for the loss of spectral information is:

[0024]

[0025] Thirdly, the present invention also provides a hyperspectral image spatial super-resolution method, applied to a trained dual-branch hyperspectral spatial super-resolution network model as described in the second aspect, comprising:

[0026] Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

[0027] Fourthly, the present invention also provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the hyperspectral image spatial super-resolution method as described in the third aspect.

[0028] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0029] 1. This invention provides a dual-branch hyperspectral spatial super-resolution network model. The super-resolution results obtained through the abundance map spatial super-resolution of the unmixed branch and the linear mixing of endmembers facilitate the utilization of spectral information, thereby improving the smoothness between spectral bands after hyperspectral spatial super-resolution reconstruction. The spatial super-resolution of the original hyperspectral image through the unmixed branch helps improve the super-resolution reconstruction effect in image space. The fusion network performs spectral upsizing on the unmixed super-resolution hyperspectral image and then concatenates and groups the upsized unmixed super-resolution hyperspectral image with the unmixed super-resolution hyperspectral image for each band, resulting in a high-resolution hyperspectral image. Therefore, compared to existing hyperspectral super-resolution techniques, this model is more conducive to the combination of spatial and spectral information, thereby improving the quality of hyperspectral image spatial super-resolution.

[0030] 2. This invention provides a dual-branch hyperspectral spatial super-resolution network model. The abundance map super-resolution module in the unmixed branch network has the same structure as the hyperspectral image super-resolution module in the non-unmixed branch network. Both improve the final super-resolution effect through multi-scale spatial super-resolution images and fusion. Existing techniques often use structures such as Transformers to improve the utilization of global spatial information, but the model has high complexity and low efficiency. The multi-scale spatial super-resolution approach improves the receptive field of the network through spatial dimensionality reduction and convolution, and the multi-scale spatial super-resolution helps to extract richer super-resolution features, thereby effectively improving the final super-resolution effect.

[0031] 3. This invention provides a hyperspectral image spatial super-resolution method, applied to a dual-branch hyperspectral spatial super-resolution network model. A low-resolution hyperspectral image is input into the dual-branch hyperspectral spatial super-resolution network model, and after a series of image processing steps, a high-resolution hyperspectral image is output. Because the abundance map super-resolution module and the hyperspectral image super-resolution module have the same structure, the overall model complexity is low, improving the efficiency of super-resolution reconstruction. Furthermore, by fully utilizing the combination of spatial and spectral information, the quality of the output high-resolution hyperspectral image is improved.

[0032] 4. This invention provides a Top-k selection unmixing method, applied to the unmixing branch network provided by this invention. It performs initial convolutional scoring on the unmixed low-resolution abundance maps and corresponding endmembers to obtain a weight score for each abundance map. Then, it selects the K low-resolution abundance maps corresponding to the highest scores using a learned adaptive threshold, resulting in the selected low-resolution abundance maps and corresponding endmembers. Existing techniques limit the unmixing accuracy by manually setting a fixed number of abundance maps and endmembers, leading to poor super-resolution performance of hyperspectral images based on unmixing. The Top-k selection unmixing method, through network training, can adaptively select an appropriate number of abundance maps and endmembers, thereby effectively improving the spatial super-resolution performance of hyperspectral images. Attached Figure Description

[0033] Figure 1 A schematic diagram of the framework of a dual-branch hyperspectral spatial super-resolution network model provided as an example of the present invention;

[0034] Figure 2 A schematic diagram of a demixing branch network provided as an example of the present invention;

[0035] Figure 3 A network diagram for multi-scale spatial super-resolution reconstruction of the abundance map super-resolution module provided in this invention example;

[0036] Figure 4 A schematic diagram of a non-dissociative branch network provided as an example of the present invention;

[0037] Figure 5 A network diagram for multi-scale spatial super-resolution reconstruction of a hyperspectral image super-resolution module provided in this invention example;

[0038] Figure 6 A schematic diagram of the fusion network after dual-branch super-resolution provided as an example of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0040] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0041] Example 1

[0042] like Figure 1As shown, the present invention provides a dual-branch hyperspectral spatial super-resolution network model product, including a dual-branch hyperspectral spatial super-resolution network model.

[0043] The dual-branch hyperspectral spatial super-resolution network model includes: a demixed branch network, a non-demixed branch network, and a fusion network;

[0044] The unmixing branch network includes an unmixing module, an abundance map super-resolution module, and a linear mixing module. The unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers. The abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map. The linear mixing module is used to linearly mix the high-resolution abundance map and endmembers to obtain an unmixed super-resolution hyperspectral image.

[0045] The unmixed branch network includes a spectral dimensionality reduction module and a hyperspectral image super-resolution module; the spectral dimensionality reduction module is used to perform spectral dimensionality reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimensionality-reduced low-resolution hyperspectral image to obtain an unmixed super-resolution hyperspectral image.

[0046] The fusion network is used to perform spectral upscaling on the undissolved super-resolution hyperspectral image, and then concatenates the upscaled undissolved super-resolution hyperspectral image with the undissolved super-resolution hyperspectral image for each band and performs grouped convolutional dimensionality reduction to obtain a high-resolution hyperspectral image.

[0047] Among them, such as Figure 1 As shown, the two-branch network shares the same input, which is a low-resolution hyperspectral image, and each branch outputs a reconstructed high-resolution image. Given a low-resolution hyperspectral image as input... The spatial resolution is h×w, and the band length is L.

[0048] like Figure 2 As shown, for the unmixing-based hyperspectral image spatial super-resolution network branch, the input is first unmixed to obtain a low-resolution abundance map. And the corresponding endmember E, where c is the number of endmembers. The unmixing module uses an adaptive Top-k selection module to obtain low-resolution abundance maps after k selections. The corresponding endmember E′ is used; after multi-scale spatial super-resolution by the abundance map super-resolution module, k high-resolution abundance maps are obtained; the high-resolution abundance maps are linearly mixed with the endmembers of the unmixing network to obtain the final unmixed super-resolution hyperspectral image.

[0049] like Figure 4As shown, for the non-unmixed hyperspectral image spatial super-resolution branch network, the spectral dimensionality reduction module is a three-layer grouped convolution; where the convolution kernel size is 3×3, the number of groups in the first convolution is L, and the output channels are L; the second convolution starts spectral dimensionality reduction, the number of groups in the convolution is L / / 2, and the output channels are L / / 2; the number of groups in the third convolution is L / / 4, and the output channels are L / / 4; where L is the number of input spectral bands, each convolution layer has Batch norm and ReLU function activation, and the super-resolution network is a multi-scale spatial super-resolution network.

[0050] Furthermore, such as Figure 6 As shown, the fusion network first upscales the output of the unmixed super-resolution network (unmixed super-resolution hyperspectral image) to L, then concatenates it with the result of the unmixed super-resolution network (unmixed super-resolution hyperspectral image) for each band, downsamples it to L through grouped convolution, and finally obtains the final output through another grouped convolution. The spectral upscaling uses bilinear interpolation. The first grouped convolution has 2L input channels, L groups, L output channels, and a 3×3 kernel size. The last convolution has L output channels, L groups, and a 3×3 kernel size. Each convolution layer has batch norm and ReLU activation.

[0051] This invention constructs a dual-branch hyperspectral spatial super-resolution network model, including a demixed hyperspectral image spatial super-resolution network branch and a non-demixed hyperspectral image spatial super-resolution network branch. The super-resolution result obtained by the abundance map spatial super-resolution and the linear mixing of endmembers in the demixed branch helps to utilize spectral information, thereby improving the smoothness between spectral bands after hyperspectral spatial super-resolution reconstruction. The original hyperspectral image spatial super-resolution in the non-demixed branch helps to improve the super-resolution reconstruction effect in image space. The fusion network concatenates the super-resolution hyperspectral images obtained from the two branch networks for each band to obtain the final high-resolution hyperspectral image. This solves the technical problems of difficulty in combining spatial and spectral information and low accuracy in high spatial super-resolution hyperspectral images. It improves the utilization rate and combination effect of spatial and spectral information, thereby improving the quality of hyperspectral image spatial super-resolution.

[0052] Based on the above embodiments, optionally, the demixing module is used to demix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers, including:

[0053] The demixing module demixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding endmembers. It also performs a preliminary convolutional scoring on the low-resolution abundance map to obtain a weight score for each abundance map. Then, it selects the K low-resolution abundance maps corresponding to the highest scores through a learned adaptive threshold to obtain the selected low-resolution abundance map and corresponding endmembers.

[0054] like Figure 2 and Figure 3 As shown, the demixing module has 4 convolutional layers with a kernel size of 3×3 and channel numbers of 8c, 4c, 2c, and c, respectively, where c is the set number of endmembers. Each convolutional layer has batch norm and ReLU activation, and the last layer uses Softmax activation to satisfy the constraint that the sum is 1. The spectrum of the input low-resolution hyperspectral image is projected onto 8c, 4c, 2c, and c sequentially through the 4 convolutional layers, resulting in c preliminary low-resolution abundance maps a and corresponding endmembers E. The low-resolution abundance map a satisfies the prior constraint that it is non-negative and its sum is 1.

[0055] For c low-resolution abundance maps a, spatial downsampling is performed twice using a 3×3 convolution kernel with a stride of 2. Then, the average and Softmax activation of each downsampled abundance map are calculated to obtain the score of each low-resolution abundance map. An adaptive threshold is used to select the k low-resolution abundance maps a′ and corresponding endmembers E′ corresponding to the highest scores, thus completing the Top-k selection.

[0056] The endmember E′ is designed with L 1×1 convolution kernels, where L is the number of bands in the hyperspectral image; convolution with a′ yields the low-resolution hyperspectral reconstructed image X. r Therefore, the following condition is met:

[0057] X r =a′E′

[0058] The difference between the reconstructed image and the input is used as the loss, and the weights of the demixing module can be obtained through training.

[0059] The low-resolution abundance map obtained after demixing is subjected to multi-scale spatial super-resolution to obtain... The spatial resolution after super-resolution is H×W. Therefore, the final unmixed super-resolution hyperspectral image Y u satisfy:

[0060] Y u =AE′

[0061] Furthermore, for the super-resolution network branch based on unmixed hyperspectral images, the input X is first downsampled to obtain... Where l is the spectral dimension after downsampling.

[0062] The downsampling method is grouped convolution to preserve the order of spectral bands. Grouped convolution, as the downsampling operation, has a downsampling factor determined by the number of groups and the output channels, reducing the number of spectra by half after each convolution. The hyperspectral image after spectral downsampling is then subjected to multi-scale spatial super-resolution processing to obtain the final hyperspectral image. The final output is then obtained through upsampling via grouped convolution. Upsampling is also performed through grouped convolution.

[0063] Furthermore, for the fusion network, the output Y of the two branches is... u and Y non-u Corresponding bands are concatenated pairwise to maintain the band order. Grouped convolution downsampling to L is then performed, followed by another layer of grouped convolution to better fuse the dual-branch super-resolution images. Therefore, the final upsampled image after fusion can be represented as:

[0064] Y = DWConv(Cat((Y non-u ,Y u ))

[0065] Where DWConv(·) represents grouped convolution and Cat(·) represents cascaded operation.

[0066] Optionally, the abundance map super-resolution module includes an encoding sub-module, a decoding sub-module, a super-resolution sub-module, and a fusion sub-module;

[0067] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

[0068] The input data is processed by two downsampling encoding modules and two upsampling decoding modules to obtain outputs of the same size. Feature maps of different sizes in the encoding and decoding parts are upsampled to the size of the super-resolution image, and the final super-resolution image is obtained through cascading and fusion.

[0069] Specifically, the given input is Where the resolution is h×w, and L is the number of features. A downsampling encoding operation is performed to obtain the encoded features. i = [1, 2, ..., n]. Satisfying:

[0070]

[0071] Where n is the number of downsampling layers, and Down(·) is the downsampling operation, using max pooling.

[0072] Each convolution consists of two group convolutions to maintain the order of spectral bands or abundance maps. The number of feature channels encoded in the intermediate layer is 4L.

[0073] Decoding operation to obtain decoding features j = [1, 2, ..., n]. Satisfying:

[0074]

[0075] Where n is the number of upsampling layers, Up(·) represents upsampling, and this invention uses bilinear interpolation. Furthermore, the decoded input... For the last encoded feature Right now Each upsampling step involves two grouped convolutions, and the number of channels in the decoding intermediate layer is 4L.

[0076] For feature layer i=[0,1,…,n] and j = [1, 2, ..., n], upsampled to the super-resolution image size to obtain super-resolution features. and satisfy:

[0077]

[0078]

[0079] For super-resolution of the features in the last decoding layer, cascaded inputs are still required. :

[0080]

[0081] The super-resolution output is obtained by concatenating and convolutionally fusing the results from various spatial scales. :

[0082]

[0083] The grouped convolution consists of two layers. The first convolution keeps the number of channels unchanged, and the second convolution reduces the number of channels to be consistent with the number of input channels.

[0084] Specifically, including:

[0085] like Figure 3As shown, the first downsampling includes two convolutions, a batch normalization layer, and a ReLU activation layer, followed by a 2×2 max pooling layer. The convolutional kernels of the convolutional layers are 3×3 in size, and the convolution method is grouped convolution. The first convolution has L input channels, L groups, and 4L output channels. The second convolution has 4L input channels, L groups, and 4L output channels.

[0086] The second downsampling includes two convolutions, a batch normalization, and an activation layer with a ReLU function, followed by a 2×2 max pooling layer. The convolutional kernels of the convolutional layers are 3×3 in size, and the convolution method is grouped convolution. The input channels of the two convolutions are 4L, the number of groups is L, and the output channels are 4L.

[0087] The first upsampling includes two convolutions, a batch norm, and an activation layer with a ReLU function, followed by bilinear interpolation with an upsampling scale of 2. The convolutional kernels of the convolutional layers are 3×3 in size, and the convolution method is grouped convolution. The input channels of the two convolutions are 4L, the number of groups is L, and the output channels are 4L.

[0088] The second upsampling includes two convolutions, a batch norm, and an activation layer with a ReLU function, followed by bilinear interpolation with an upsampling scale of 2. The convolutional kernels of the convolutional layers are 3×3 in size, and the convolution method is grouped convolution. The input of the first convolution is concatenated with the feature map after the first downsampling. The input channels of the convolution are 8L, the number of groups is L, and the output channels are 4L. The input channels of the second convolution are 4L, the number of groups is L, and the output channels are L.

[0089] The feature map with the smallest size is upsampled to the super-resolution size, including a single convolution, a batch norm, and an activation layer with a ReLU function, and finally connected to a bilinear interpolation with an upsampling scale of 4K. The size of the convolution kernel is 3×3, and the convolution method is grouped convolution. The input channels of the convolution are 4L, the number of groups is L, the output channels are L, and K is the super-resolution factor.

[0090] The feature map of the intermediate size is upsampled to the super-resolution size, including a single convolution, a batch norm and a ReLU activation layer, and finally a bilinear interpolation with an upsampling scale of 2K. The convolution kernel size is 3×3, and the convolution method is grouped convolution. The input channels of the convolution are 4L, the number of groups is L, the output channels are L, and K is the super-resolution factor.

[0091] The feature maps of the largest size are concatenated and then upsampled to the super-resolution size, including a single convolution, a Batchnorm activation layer, and a ReLU function activation layer, and finally a bilinear interpolation with an upsampling scale of K. The convolution kernel size is 3×3, and the convolution method is grouped convolution. The input channels of the convolution are 2L, the number of groups is L / / 4, the output channel is L, and K is the super-resolution factor.

[0092] The cascaded super-resolution feature maps are then convolved, including two convolutions, batch normalization, and ReLU activation layers. The first convolution has a 3×3 kernel and uses a grouped convolution method. The input channels are 3L, the number of groups is L, and the output channels are 3L. The second convolution also has a 3×3 kernel and uses a grouped convolution method. The input channels are 3L, the number of groups is L, and the output channels are L.

[0093] Optionally, the hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule, and a fusion submodule;

[0094] The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map; the super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps; the fusion submodule is used to fuse high-resolution feature maps of the same size to obtain an unmixed super-resolution hyperspectral image.

[0095] The hyperspectral image super-resolution module and the abundance map super-resolution module have the same structure and similar functions, and will not be described in detail here. Their structure and functions are referenced elsewhere. Figure 5 And the description in the above abundance map super-resolution module.

[0096] Based on the above embodiments, the present invention also provides a training method for a dual-branch hyperspectral spatial super-resolution network model, wherein the dual-branch hyperspectral spatial super-resolution network model of any of the dual-branch hyperspectral spatial super-resolution network models described in any of the above embodiments is trained to obtain a trained dual-branch hyperspectral spatial super-resolution network model.

[0097] Training loss includes spectral information loss and spatial information loss;

[0098] The spatial information loss includes the spatial L1 loss of the super-resolution image and the spatial L1 loss of the demixing reconstruction; the expression for the spatial information loss is:

[0099]

[0100] Wherein, Y1 is the upscaled, unmixed super-resolution hyperspectral image, Y2 is the unmixed super-resolution hyperspectral image, and Y3 is the high-resolution hyperspectral image. For a true high-resolution hyperspectral image, X is the input low-resolution hyperspectral image, X r The selected low-resolution abundance map and the reconstructed low-resolution hyperspectral image after linear mixing of the corresponding endmembers;

[0101] The spectral information loss includes the spectral information loss L of the super-resolution image. sam The spectral information loss L during unmixing reconstruction sam The expression for the loss of spectral information is:

[0102]

[0103] The loss function includes spatial information loss and spectral information loss; the spatial information loss includes the reconstruction spatial information loss in the unmixing branch and the super-resolution image spatial information loss in the unmixing branch, while the spectral information loss includes the spectral loss in the unmixing branch and the super-resolution image spectral loss in the unmixing branch. Therefore, the final loss is: L all =L spectral +L spatial .

[0104] Example 2

[0105] This invention also provides a hyperspectral image spatial super-resolution method, which applies a trained bi-branch hyperspectral spatial super-resolution network model as described in any one of the embodiments, including:

[0106] Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

[0107] The low-resolution hyperspectral image is input into the dual-branch hyperspectral spatial super-resolution network model provided in this embodiment. After a series of image processing steps, a high-resolution hyperspectral image is output. Since the abundance map super-resolution module and the hyperspectral image super-resolution module have the same structure, the overall model complexity is low, improving the efficiency of super-resolution reconstruction. Furthermore, by fully utilizing the combination of spatial and spectral information, the quality of the output high-resolution hyperspectral image is improved.

[0108] The hyperspectral image spatial super-resolution method provided in this embodiment of the invention is based on the trained dual-branch hyperspectral spatial super-resolution network model provided in any embodiment of the invention, and has corresponding beneficial effects.

[0109] Example 3

[0110] The present invention also provides a machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the hyperspectral image spatial super-resolution method as described in Embodiment 2.

[0111] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A device for a dual-branch hyperspectral spatial super-resolution network model, characterized in that, Including a dual-branch hyperspectral spatial super-resolution network model; The dual-branch hyperspectral spatial super-resolution network model includes: a demixed branch network, a non-demixed branch network, and a fusion network; The unmixing branch network includes an unmixing module, an abundance map super-resolution module, and a linear mixing module. The unmixing module is used to unmix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers. The abundance map super-resolution module is used to perform multi-scale spatial super-resolution on the low-resolution abundance map to obtain a high-resolution abundance map. The linear mixing module is used to linearly mix the high-resolution abundance map and endmembers to obtain an unmixed super-resolution hyperspectral image. The unmixed branch network includes a spectral dimensionality reduction module and a hyperspectral image super-resolution module; the spectral dimensionality reduction module is used to perform spectral dimensionality reduction on the input low-resolution hyperspectral image; the hyperspectral image super-resolution module is used to perform multi-scale spatial super-resolution on the dimensionality-reduced low-resolution hyperspectral image to obtain an unmixed super-resolution hyperspectral image. The fusion network is used to perform spectral upsizing on the undissolved super-resolution hyperspectral image, and then perform cascading and grouped convolution dimensionality reduction on the upsized undissolved super-resolution hyperspectral image and the undissolved super-resolution hyperspectral image for each band to obtain a high-resolution hyperspectral image. The demixing module is used to demix the input low-resolution hyperspectral image to obtain a low-resolution abundance map and endmembers. The demixing module demixes the input low-resolution hyperspectral image to obtain a preliminary low-resolution abundance map and corresponding endmembers, performs preliminary convolution scoring on the low-resolution abundance maps to obtain a weight score for each abundance map, and then selects the K low-resolution abundance maps corresponding to the highest scores through a learned adaptive threshold to obtain the selected low-resolution abundance map and corresponding endmembers.

2. The apparatus as claimed in claim 1, characterized in that, The abundance map super-resolution module includes an encoding sub-module, a decoding sub-module, a super-resolution sub-module, and a fusion sub-module; The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map. The super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps. The fusion submodule is used to fuse high-resolution feature maps of the same size to obtain a high-resolution abundance map.

3. The apparatus as described in claim 1, characterized in that, The hyperspectral image super-resolution module includes an encoding submodule, a decoding submodule, a super-resolution submodule, and a fusion submodule; The encoding submodule is used to downsample the input data twice to obtain an encoded feature map; the decoding submodule is used to upsample the encoded feature map twice to obtain a decoded feature map. The super-resolution submodule is used to perform multi-scale spatial super-resolution on the encoded feature map and the decoded feature map to obtain multiple high-resolution feature maps. The fusion submodule is used to fuse high-resolution feature maps of the same size to obtain an unmixed super-resolution hyperspectral image.

4. A training method for a dual-branch hyperspectral spatial super-resolution network model, characterized in that, The dual-branch hyperspectral spatial super-resolution network model in the device of any one of claims 1-3 is trained to obtain the trained dual-branch hyperspectral spatial super-resolution network model. Training loss includes spectral information loss and spatial information loss; The spatial information loss includes the spatial information of the super-resolution image. Loss and unmixed reconstruction space Loss; the expression for the spatial information loss is: in, This is the upgraded, unmixed, super-resolution hyperspectral image. To unmix super-resolution hyperspectral images, For high-resolution hyperspectral images, For true high-resolution hyperspectral images, For the input low-resolution hyperspectral image, The low-resolution hyperspectral image is reconstructed by linear mixing of the selected low-resolution abundance map and the corresponding endmembers. The spectral information loss includes the spectral information loss of the super-resolution image. Loss of spectral information during reconstruction and unmixing The expression for the loss of spectral information is: 。 5. A spatial super-resolution method for hyperspectral images, characterized in that, The application of the trained dual-branch hyperspectral spatial super-resolution network model as described in claim 4 includes: Input a low-resolution hyperspectral image and output a high-resolution hyperspectral image.

6. A machine-readable storage medium storing machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the hyperspectral image spatial super-resolution method as described in claim 5.

Images