High spatial resolution hyperspectral image generation method, device, equipment and medium

Abstract

The present application relates to the field of image processing, and provides a high spatial resolution hyperspectral image generation method, device, equipment and medium, the method comprises: carrying out down-sampling processing to input low spatial resolution hyperspectral image and input high spatial resolution multispectral image, and obtaining output low spatial resolution multispectral image; based on the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image, the input high spatial resolution multispectral image is carried out spectral super-resolution processing, and the simulated high spatial resolution hyperspectral image is obtained;The fusion image is carried out down-sampling processing, and output low spatial resolution hyperspectral image and output high spatial resolution multispectral image are obtained;The fusion image is obtained by fusing the simulated high spatial resolution hyperspectral image.The present application realizes the high-quality reconstruction of spectral image by three-stage unsupervised reversible neural network architecture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a high spatial resolution hyperspectral image generation method, device, equipment and medium. BACKGROUND

[0002] At present, the high spatial resolution hyperspectral image generation method based on spectral fusion can be roughly divided into two categories: traditional method and deep learning based method. However, the traditional method depends on a large amount of prior knowledge, so the generalization ability of the model is limited. The deep learning based method can be divided into supervised learning and unsupervised learning. The supervised deep learning fusion method cannot obtain a usable training set for training, which limits the application. The model of the unsupervised deep learning method is mainly based on a convolutional neural network with back propagation to fuse or generate images, which often causes the high-frequency information in the image to be irreversibly compressed or lost. SUMMARY

[0003] The present application provides a high spatial resolution hyperspectral image generation method, device, equipment and medium to solve the problem of loss of high-frequency information in the existing high spatial resolution hyperspectral image generation method.

[0004] The present application provides a high spatial resolution hyperspectral image generation method, which comprises the following steps:

[0005] The three-stage unsupervised reversible neural network architecture based on image prior knowledge comprises a prior information mining network, a spectral mapping network and a feature fusion network. The three-stage unsupervised reversible neural network architecture is trained based on a low spatial resolution hyperspectral image sample and a high spatial resolution multispectral image sample;

[0006] The prior information mining network is used to perform down-sampling processing on the input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image to obtain an output low spatial resolution multispectral image;

[0007] The spectral mapping network, the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image are used to perform spectral super-resolution processing on the input high spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image;

[0008] The feature fusion network is used to perform down-sampling processing on the fused image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image. The fused image is obtained by fusing the simulated high spatial resolution hyperspectral image.

[0009] According to the high spatial resolution hyperspectral image generation method provided by the present application, the high spatial resolution hyperspectral image generation method further comprises:

[0010] generate a first low spatial resolution multispectral image based on the low spatial resolution hyperspectral image sample and a first learnable weight function;

[0011] generate a second low spatial resolution multispectral image based on the high spatial resolution multispectral image sample and a second learnable weight function; the low spatial resolution hyperspectral image sample and the high spatial resolution multispectral image sample are scene-registered images;

[0012] determine a first loss function based on a consistency constraint of the first low spatial resolution multispectral image and the second low spatial resolution multispectral image;

[0013] construct the prior information mining network based on the first loss function.

[0014] According to the high spatial resolution hyperspectral image generation method provided by the application, the high spatial resolution hyperspectral image generation method further comprises:

[0015] perform spectral super-resolution processing on the input low spatial resolution multispectral image to obtain a target low spatial resolution hyperspectral image; the input low spatial resolution multispectral image comprises the first low spatial resolution multispectral image and the second low spatial resolution multispectral image;

[0016] determine a second loss function based on a constraint of the low spatial resolution hyperspectral image sample and the target low spatial resolution hyperspectral image;

[0017] construct the spectral mapping network based on the second loss function.

[0018] According to the high spatial resolution hyperspectral image generation method provided by the application, the high spatial resolution hyperspectral image generation method further comprises:

[0019] fuse the high spatial resolution hyperspectral image sample obtained based on the spectral mapping network to obtain a fused sample;

[0020] perform down-sampling processing on the fused sample based on the input low spatial resolution multispectral image to obtain a down-sampled image;

[0021] determine a third loss function based on a similarity between an input sample and the down-sampled image;

[0022] construct the feature fusion network based on the third loss function.

[0023] According to the high spatial resolution hyperspectral image generation method provided by the application, the high spatial resolution hyperspectral image generation method further comprises:

[0024] tensorizing the simulated high spatial resolution hyperspectral image;

[0025] inputting the spliced result into a reversible neural network layer to obtain the fused image.

[0026] According to the high spatial resolution hyperspectral image generation method provided by the application, the feature fusion network is used for performing down-sampling processing on the fused image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image.

[0027] The feature fusion network is used for performing down-sampling processing on the fused image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image based on the output low spatial resolution multispectral image.

[0028] The application further provides a high spatial resolution hyperspectral image generation device, which comprises the following modules.

[0029] The prior information mining module is used for performing down-sampling processing on the input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image to obtain an output low spatial resolution multispectral image.

[0030] The spectral mapping module is used for performing spectral super-resolution processing on the input high spatial resolution multispectral image based on the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image.

[0031] The feature fusion module is used for performing down-sampling processing on the fused image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image. The fused image is obtained by performing fusion processing on the simulated high spatial resolution hyperspectral image.

[0032] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the high spatial resolution hyperspectral image generation method is realized.

[0033] The application further provides a non-transitory computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the high spatial resolution hyperspectral image generation method is realized.

[0034] The application further provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the high spatial resolution hyperspectral image generation method is realized.

[0035] The application provides a high spatial resolution hyperspectral image generation method, device, equipment and medium, which is realized based on a three-stage unsupervised reversible neural network architecture including a prior information mining network, a spectral mapping network and a feature fusion network, the prior information mining network is used for performing down-sampling processing on an input spectral image to obtain an output low spatial resolution multispectral image, the spectral mapping network is used for performing spectral super-resolution processing on the multispectral image to obtain a simulated high spatial resolution hyperspectral image, and the feature fusion network is used for performing down-sampling processing on a fusion image to extract detailed features, so that lossless transmission of spectral image information is realized, and high-frequency information features of the original image are reserved. The three-stage unsupervised reversible neural network architecture is used for high-quality reconstruction of the spectral image. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0037] Figure 1 is one of the flowcharts of the high spatial resolution hyperspectral image generation method provided by the present application.

[0038] Figure 2 is another flowchart of the high spatial resolution hyperspectral image generation method provided by the present application.

[0039] Figure 3 is a three-stage unsupervised reversible neural network architecture based on image prior knowledge provided by the present application.

[0040] Figure 4 is a feature fusion network provided by the present application.

[0041] Figure 5 is a structural schematic diagram of the high spatial resolution hyperspectral image generation device provided by the present application.

[0042] Figure 6 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort belong to the protection scope of the present application.

[0044] The application is described below Figures 1-6 The application provides a high spatial resolution hyperspectral image generation method, device, equipment and medium.

[0045] Figure 1 is one of the flowcharts of the high spatial resolution hyperspectral image generation method provided by the application, as Figure 1 shown, the method comprises the following:

[0046] The three-stage unsupervised reversible neural network architecture based on image prior knowledge comprises a prior information mining network, a spectral mapping network and a feature fusion network; the three-stage unsupervised reversible neural network architecture is trained based on a low spatial resolution hyperspectral image sample and a high spatial resolution multispectral image sample;

[0047] Step 1000, performing down-sampling processing on the input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image based on the prior information mining network to obtain an output low spatial resolution multispectral image;

[0048] The high spatial resolution hyperspectral image generation method provided by the application is realized through the three-stage unsupervised reversible neural network architecture based on image prior knowledge. The three-stage unsupervised reversible neural network architecture based on image prior knowledge comprises three stages, namely a prior information mining stage (i.e., the prior information mining network), a spectral channel mapping stage (i.e., the spectral mapping network) and a detail feature fusion stage (i.e., the feature fusion network). The low spatial resolution hyperspectral image (i.e., the input low spatial resolution hyperspectral image in the embodiment) and the high spatial resolution multispectral image (i.e., the input high spatial resolution multispectral image in the embodiment) are input into the prior information mining network, and the input is subjected to down-sampling processing by the prior information mining network to obtain a low spatial resolution multispectral image, i.e., the output low spatial resolution multispectral image in the embodiment.

[0049] Step 2000, performing spectral super-resolution processing on the input high spatial resolution multispectral image based on the spectral mapping network, the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image;

[0050] After the training of the spectral mapping network is completed, two consistent spectral super-resolution networks are used to perform spectral super-resolution processing on the two low spatial resolution multispectral images generated by the prior information mining network, to obtain low spatial resolution hyperspectral images again. Then, the two low spatial resolution hyperspectral images obtained by the spectral super-resolution processing are constrained with the input low spatial resolution hyperspectral image. After the constraint is completed, the input high spatial resolution multispectral image is super-resolution processed by the two spectral super-resolution networks through weight sharing, to obtain two simulated high spatial resolution hyperspectral images.

[0051] Step 3000, performing down-sampling processing on the fusion image based on the feature fusion network to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; the fusion image is obtained by performing fusion processing on the simulated high spatial resolution hyperspectral image.

[0052] In the third stage, the two simulated high spatial resolution hyperspectral images obtained by the spectral mapping network are fused to obtain a fusion image. Then, the fusion image is down-sampled to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image. In the third stage, a reversible neural network is used to extract the detailed features of the fusion image, so that the lossless information transmission is realized, and the high-frequency information features of the original image are greatly retained.

[0053] In this embodiment, based on the three-stage unsupervised reversible neural network architecture including the prior information mining network, the spectral mapping network and the feature fusion network, the input spectral image is down-sampled by the prior information mining network to obtain an output low spatial resolution multispectral image; the multispectral image is super-resolution processed by the spectral mapping network to obtain a simulated high spatial resolution hyperspectral image; and the fusion image is down-sampled by the feature fusion network to extract detailed features, so that the lossless transmission of spectral image information is realized, and the high-frequency information features of the original image are retained. The three-stage unsupervised reversible neural network architecture is used to realize the high-quality reconstruction of the spectral image.

[0054] Figure 2 is a flowchart of the method for generating a high spatial resolution hyperspectral image provided by the application, as shown in Figure 2 The method can further include:

[0055] Step 10, generating a first low spatial resolution multispectral image based on the low spatial resolution hyperspectral image sample and a first learnable weight function;

[0056] Step 20, generating a second low spatial resolution multi-spectral image based on the high spatial resolution multi-spectral image sample and a second learnable weight function; the low spatial resolution hyper-spectral image sample and the high spatial resolution multi-spectral image sample are scene-registered images;

[0057] Step 30, determining a first loss function based on a consistency constraint of the first low spatial resolution multi-spectral image and the second low spatial resolution multi-spectral image;

[0058] Step 40, constructing the prior information mining network based on the first loss function.

[0059] As shown in Figure 3 , the prior information mining network includes a spatial prior block and a spectral prior block , as shown in formula 1. The spatial prior block and the spectral prior block are mainly used to learn the spectral response function (SRF) and the point spread function (PSF) prior information needed for spatial down-sampling and spectral down-sampling of the image. The high spatial resolution multi-spectral image is spatially down-sampled to obtain a low spatial resolution multi-spectral image, and the low spatial resolution hyper-spectral image is spectrally down-sampled to obtain a low spatial resolution multi-spectral image.

[0060] ; (1)

[0061] Among them, the spatial prior block and the spectral prior block can be a learnable weight function, that is, the first learnable weight function and the second learnable weight function in the embodiment.

[0062] In the training stage of the prior information mining network, the input low spatial resolution hyper-spectral image (LrHSI) sample, that is, the in formula 1 above, and the input high spatial resolution multi-spectral image (HrMSI) sample, that is, the in formula 1 above, are highly registered image samples of the same scene. Therefore, the two generated low spatial resolution multi-spectral image samples and , that is, the first low spatial resolution multi-spectral image and the second low spatial resolution multi-spectral image in the embodiment, should be consistent, and the consistency condition is used for constraint to obtain the loss function of the first-stage prior information mining network (i.e., the first loss function in this embodiment), as shown in Formula 2.

[0063] (2)

[0064] in, The number of data points in the 3D image is represented by the first loss function, which is used to constrain the absolute error between the first low spatial resolution multispectral image sample and the second low spatial resolution multispectral image sample, thus ensuring the consistency of the two generated images.

[0065] This embodiment learns the SRF and PSF prior information needed for spatial and spectral downsampling of images through a prior information mining network.

[0066] In one embodiment, the high spatial resolution hyperspectral image generation method provided by this invention may further include:

[0067] Step 50: Perform spectral super-resolution processing on the input low spatial resolution multispectral image to obtain the target low spatial resolution hyperspectral image; the input low spatial resolution multispectral image includes the first low spatial resolution multispectral image and the second low spatial resolution multispectral image;

[0068] Step 60: Determine the second loss function based on the constraints of the low spatial resolution hyperspectral image samples and the target low spatial resolution hyperspectral image;

[0069] Step 70: Construct the spectral mapping network based on the second loss function.

[0070] Specifically, in the second stage of training the spectral mapping network, the image fusion problem is reduced back to a spectral mapping problem. For example... Figure 3 As shown, two consistent spectral super-resolution networks are used to perform spectral super-resolution processing on the two low spatial resolution multispectral image samples generated in the first stage (i.e., the input low spatial resolution multispectral image in this embodiment) to obtain a new low spatial resolution hyperspectral image (i.e., the target low spatial resolution hyperspectral image in this embodiment), as shown in Formula 3.

[0071] (3)

[0072] (4)

[0073] (5)

[0074] in, and The spectral super-resolution network is shown. Two images obtained by spectral super-resolution processing are constrained with the initial input low spatial resolution hyperspectral image to obtain a loss function of the second stage (i.e., the second loss function in the embodiment) as shown in formula 4.

[0075] The spectral mapping network of the second stage is trained by using the above process. After the training is completed, the input high spatial resolution multispectral image is super-resolution processed by using the two spectral super-resolution networks through weight sharing to obtain two simulated high spatial resolution hyperspectral images as shown in formula 5.

[0076] The embodiment re-obtains a low spatial resolution hyperspectral image by performing spectral super-resolution processing on the two low spatial resolution multispectral images generated by the prior information mining network through the spectral mapping network.

[0077] In one embodiment, the high spatial resolution hyperspectral image generation method provided by the embodiment can further include:

[0078] Step 80, fusing the high spatial resolution hyperspectral image samples obtained based on the spectral mapping network to obtain fused samples;

[0079] Step 90, performing down-sampling processing on the fused samples based on the input low spatial resolution multispectral image to obtain a down-sampled image;

[0080] Step 100, determining a third loss function based on the similarity between the input samples and the down-sampled image;

[0081] Step 110, constructing the feature fusion network based on the third loss function.

[0082] Specifically, as shown in Figure 3 and Figure 4 , the high-frequency features of the fused image obtained by fusing the two simulated high spatial resolution hyperspectral images are extracted to realize the transmission of the lossless information of the spectral image in the training stage of the feature fusion network. First, the two simulated HrHSI obtained by the second stage spectral mapping network are fused, and then the high-frequency detail features of the fused image are extracted.

[0083] Referring to formula 6, wherein, is tensor splicing; is the output result of the first reversible neural network layer, and the input and output results of the first reversible network layer can be obtained as shown in formula 7.

[0084] ; (6)

[0085] ; (7)​

[0086] (8)

[0087] (9)

[0088] (10)

[0089] Therefore, through After processing through several reversible neural networks, the final fusion result is obtained, as shown in Equation 8. Then, using the SRF and PSF downsampling information learned in the first stage, the final fusion result is obtained. Downsampling is performed to obtain a downsampled image, as shown in Equation 9. The similarity between the downsampled image and the input image is calculated to constrain the model. The fusion quality is used to obtain the loss function of the third-stage feature fusion network (i.e., the third loss function in this embodiment), as shown in Formula 10.

[0090] This embodiment uses a reversible neural network to extract detailed features from the fused image, achieving lossless information transmission and greatly preserving the high-frequency information features of the original image.

[0091] In one embodiment, the high spatial resolution hyperspectral image generation method provided by this invention may further include:

[0092] Step 3100: Perform tensor stitching on the simulated high spatial resolution hyperspectral image;

[0093] Step 3200: Input the splicing result into the reversible neural network layer to obtain the fused image.

[0094] Specifically, such as Figure 4 As shown, This is the output of the first reversible neural network layer stack, i.e. Figure 4 In In formulas 5 and 6 above... and The simulated high spatial resolution hyperspectral image in this embodiment is then subjected to tensor stitching using Formula 6 to obtain the stitching result. The concatenated result is input into a multi-layer reversible neural network to obtain the output results of each reversible neural network layer. arrive That is, the fused image in this embodiment. .

[0095] This embodiment obtains the final fusion result through multiple reversible neural networks.

[0096] In one embodiment, the high spatial resolution hyperspectral image generation method provided by the embodiment of the present application can further include:

[0097] Step 3300, performing down-sampling processing on the fusion image based on the output low spatial resolution multispectral image, to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image.

[0098] Specifically, the SRF and PSF down-sampling information (i.e., the output low spatial resolution multispectral image) learned by the first-stage prior information mining network is used to perform down-sampling processing on the fusion image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image.

[0099] The embodiment extracts the detail features of the fusion image through the reversible neural network, realizes lossless information transmission, and greatly retains the high-frequency information features of the original image.

[0100] The high spatial resolution hyperspectral image generation device provided by the present application is described below, and the high spatial resolution hyperspectral image generation device described below can be correspondingly referred to the high spatial resolution hyperspectral image generation method described above.

[0101] Please refer to Figure 5 , the present application also provides a high spatial resolution hyperspectral image generation device, comprising:

[0102] A prior information mining module 301 is configured to perform down-sampling processing on the input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image, to obtain an output low spatial resolution multispectral image.

[0103] A spectral mapping module 302 is configured to perform spectral super-resolution processing on the input high spatial resolution multispectral image based on the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image, to obtain an analog high spatial resolution hyperspectral image.

[0104] A feature fusion module 303 is configured to perform down-sampling processing on a fusion image, to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; the fusion image is obtained by performing fusion processing on the analog high spatial resolution hyperspectral image.

[0105] Optionally, the high spatial resolution hyperspectral image generation device further comprises:

[0106] A first low spatial resolution multispectral image generation module is configured to generate a first low spatial resolution multispectral image based on the low spatial resolution hyperspectral image sample and a first learnable weight function.

[0107] a second low-spatial-resolution multispectral image generation module configured to generate a second low-spatial-resolution multispectral image based on the high-spatial-resolution multispectral image sample and a second learnable weight function; the low-spatial-resolution hyperspectral image sample and the high-spatial-resolution multispectral image sample are scene-registered images;

[0108] a first loss function determination module configured to determine a first loss function based on a consistency constraint of the first low-spatial-resolution multispectral image and the second low-spatial-resolution multispectral image;

[0109] a priori information mining network construction module configured to construct the a priori information mining network based on the first loss function.

[0110] Optionally, the high-spatial-resolution hyperspectral image generation apparatus further includes:

[0111] a target low-spatial-resolution hyperspectral image determination module configured to perform spectral super-resolution processing on an input low-spatial-resolution multispectral image to obtain a target low-spatial-resolution hyperspectral image; the input low-spatial-resolution multispectral image includes the first low-spatial-resolution multispectral image and the second low-spatial-resolution multispectral image;

[0112] a second loss function determination module configured to determine a second loss function based on a constraint of the low-spatial-resolution hyperspectral image sample and the target low-spatial-resolution hyperspectral image;

[0113] a spectral mapping network construction module configured to construct the spectral mapping network based on the second loss function.

[0114] Optionally, the high-spatial-resolution hyperspectral image generation apparatus further includes:

[0115] a fusion module configured to fuse the high-spatial-resolution hyperspectral image sample obtained based on the spectral mapping network to obtain a fused sample;

[0116] a down-sampling processing module configured to perform down-sampling processing on the fused sample based on the input low-spatial-resolution multispectral image to obtain a down-sampled image;

[0117] a third loss function determination module configured to determine a third loss function based on a similarity between an input sample and the down-sampled image;

[0118] a feature fusion network construction module configured to construct the feature fusion network based on the third loss function.

[0119] Optionally, the high-spatial-resolution hyperspectral image generation apparatus further includes:

[0120] a tensor splicing module configured to splice the simulated high-spatial-resolution hyperspectral image in a tensor manner;

[0121] a fusion image determining module configured to input the splicing result into a reversible neural network layer to obtain the fusion image.

[0122] Optionally, the feature fusion module comprises:

[0123] a fusion image down-sampling processing unit configured to perform down-sampling processing on the fusion image based on the output low-spatial-resolution multispectral image to obtain an output low-spatial-resolution hyperspectral image and an output high-spatial-resolution multispectral image.

[0124] Figure 4 An example of a schematic diagram of a physical structure of an electronic device is shown in Figure 4 As shown in the figure, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can complete mutual communication through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute a high-spatial-resolution hyperspectral image generation method, which includes: performing down-sampling processing on an input low-spatial-resolution hyperspectral image and an input high-spatial-resolution multispectral image to obtain an output low-spatial-resolution multispectral image; performing spectral super-resolution processing on the input high-spatial-resolution multispectral image based on the input low-spatial-resolution hyperspectral image and the output low-spatial-resolution multispectral image to obtain a simulated high-spatial-resolution hyperspectral image; performing down-sampling processing on a fusion image to obtain an output low-spatial-resolution hyperspectral image and an output high-spatial-resolution multispectral image; the fusion image is obtained by performing fusion processing on the simulated high-spatial-resolution hyperspectral image.

[0125] Further, the logic instructions in the memory 430 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various media that can store program codes.

[0126] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the high spatial resolution hyperspectral image generation method provided by the above-mentioned methods. The method comprises: performing down-sampling processing on an input low spatial resolution hyperspectral image and an input high spatial resolution multispectral image to obtain an output low spatial resolution multispectral image; performing spectral super-resolution processing on the input high spatial resolution multispectral image based on the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image; and performing down-sampling processing on a fusion image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; the fusion image is obtained by performing fusion processing on the simulated high spatial resolution hyperspectral image.

[0127] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the high spatial resolution hyperspectral image generation method provided by the above-mentioned methods. The method comprises: performing down-sampling processing on an input low spatial resolution hyperspectral image and an input high spatial resolution multispectral image to obtain an output low spatial resolution multispectral image; performing spectral super-resolution processing on the input high spatial resolution multispectral image based on the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image; and performing down-sampling processing on a fusion image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; the fusion image is obtained by performing fusion processing on the simulated high spatial resolution hyperspectral image.

[0128] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0129] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0130] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of high spatial resolution hyperspectral image generation, characterized by, The method comprises the following steps: The three-stage unsupervised reversible neural network architecture based on image prior knowledge comprises a prior information mining network, a spectral mapping network and a feature fusion network; the three-stage unsupervised reversible neural network architecture is trained based on low spatial resolution hyperspectral image samples and high spatial resolution multispectral image samples; The input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image are subjected to down-sampling processing based on the prior information mining network, to obtain an output low spatial resolution multispectral image; The spectral super-resolution processing is performed on the input high spatial resolution multispectral image based on the spectral mapping network, the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image, to obtain a simulated high spatial resolution hyperspectral image; The down-sampling processing is performed on a fusion image based on the feature fusion network, to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; The fusion image is obtained by performing fusion processing on the simulated high spatial resolution hyperspectral image; Wherein: The first low spatial resolution multispectral image is generated based on the low spatial resolution hyperspectral image samples and a first learnable weight function; the second low spatial resolution multispectral image is generated based on the high spatial resolution multispectral image samples and a second learnable weight function; the low spatial resolution hyperspectral image samples and the high spatial resolution multispectral image samples are scene-registered images; the first loss function is determined based on the consistency constraint of the first low spatial resolution multispectral image and the second low spatial resolution multispectral image; the prior information mining network is constructed based on the first loss function; The spectral super-resolution processing is performed on the input low spatial resolution multispectral image, to obtain a target low spatial resolution hyperspectral image; the input low spatial resolution multispectral image comprises the first low spatial resolution multispectral image and the second low spatial resolution multispectral image; the second loss function is determined based on the constraint of the low spatial resolution hyperspectral image samples and the target low spatial resolution hyperspectral image; the spectral mapping network is constructed based on the second loss function; The fusion sample is obtained by fusing the high spatial resolution hyperspectral image sample obtained based on the spectral mapping network; the down-sampling processing is performed on the fusion sample based on the input low spatial resolution multispectral image, to obtain a down-sampling image; the third loss function is determined based on the similarity between the input sample and the down-sampling image; the feature fusion network is constructed based on the third loss function.

2. The high spatial resolution hyperspectral image generation method of claim 1, wherein, The method for generating a high spatial resolution hyperspectral image further comprises: The simulated high spatial resolution hyperspectral image is subjected to tensor splicing; The splicing result is input into a reversible neural network layer, to obtain the fusion image.

3. The high spatial resolution hyperspectral image generation method of claim 1, wherein, The down-sampling processing of the fusion image based on the feature fusion network to obtain the output low spatial resolution hyperspectral image and the output high spatial resolution multispectral image comprises: Based on the output low spatial resolution multispectral image, a fusion image is down-sampling processed to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image.

4. A high spatial resolution hyperspectral image generation apparatus characterized by comprising: Comprise: The three-stage unsupervised reversible neural network architecture based on image prior knowledge comprises a prior information mining network, a spectral mapping network and a feature fusion network; the three-stage unsupervised reversible neural network architecture is trained based on low spatial resolution hyperspectral image samples and high spatial resolution multispectral image samples; The prior information mining module is configured to perform down-sampling processing on the input low spatial resolution hyperspectral image and the input high spatial resolution multispectral image based on the prior information mining network to obtain an output low spatial resolution multispectral image; The spectral mapping module is configured to perform spectral super-resolution processing on the input high spatial resolution multispectral image based on the spectral mapping network, the input low spatial resolution hyperspectral image and the output low spatial resolution multispectral image to obtain a simulated high spatial resolution hyperspectral image; The feature fusion module is configured to perform down-sampling processing on a fusion image to obtain an output low spatial resolution hyperspectral image and an output high spatial resolution multispectral image; The fusion image is obtained by fusing the simulated high spatial resolution hyperspectral image; Wherein: based on the low spatial resolution hyperspectral image samples and a first learnable weight function, a first low spatial resolution multispectral image is generated; based on the high spatial resolution multispectral image samples and a second learnable weight function, a second low spatial resolution multispectral image is generated; the low spatial resolution hyperspectral image samples and the high spatial resolution multispectral image samples are scene-registered images; based on the consistency constraint of the first low spatial resolution multispectral image and the second low spatial resolution multispectral image, a first loss function is determined; the prior information mining network is constructed based on the first loss function; Spectral super-resolution processing is performed on an input low spatial resolution multispectral image to obtain a target low spatial resolution hyperspectral image; the input low spatial resolution multispectral image comprises the first low spatial resolution multispectral image and the second low spatial resolution multispectral image; based on the constraint of the low spatial resolution hyperspectral image samples and the target low spatial resolution hyperspectral image, a second loss function is determined; the spectral mapping network is constructed based on the second loss function; The high spatial resolution hyperspectral image samples obtained based on the spectral mapping network are fused to obtain a fusion sample; the fusion sample is down-sampling processed based on the input low spatial resolution multispectral image to obtain a down-sampling image; based on the similarity between the input sample and the down-sampling image, a third loss function is determined; the feature fusion network is constructed based on the third loss function.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the high spatial resolution hyperspectral image generation method according to any one of claims 1 to 3. 6.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the high spatial resolution hyperspectral image generation method according to any one of claims 1 to 3.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the high spatial resolution hyperspectral image generation method according to any one of claims 1 to 3.

Images