Hyperspectral image reconstruction method, device, equipment and medium based on encoding and decoding
By constructing a hyperspectral image reconstruction network, encoding, initial reconstruction, depth feature extraction and noise removal, the problem of poor recovery quality of hyperspectral image is solved, and efficient and low-cost hyperspectral image reconstruction effect is achieved.
Patent Information
- Application Number
- CN202411059685.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-05
AI Technical Summary
Among the existing hyperspectral image reconstruction methods, the hyperspectral image restoration quality is poor, which limits its application in low resources and other fields.
A hyperspectral image reconstruction method based on encoding and decoding is proposed. By constructing a hyperspectral image reconstruction network, including encoding network, initial reconstruction network and deep reconstruction network, it performs compression encoding, initial reconstruction, deep feature extraction and noise removal, and improves image reconstruction quality.
By combining the variable Bayesian loss function under deep learning method, the data advantages are used to improve the quality of hyperspectral image reconstruction, and high-quality data acquisition is achieved while reducing the cost of high-resolution cameras.
Smart Images

Figure CN118570062B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hyperspectral image reconstruction, and in particular to a method, device, equipment and medium for hyperspectral image reconstruction based on coding and decoding. Background Art
[0002] Hyperspectral images are rich in a large amount of spatial and spectral information of the target. By analyzing the spectral information, materials can be accurately distinguished with a higher degree of confidence. Therefore, hyperspectral imaging can be widely used in remote sensing, geological exploration, industrial monitoring, medical diagnosis, military and other fields. However, traditional hyperspectral cameras are expensive, and some applications have a heavy burden of data acquisition and processing, which limits their application in low-resource fields. Computational hyperspectral imaging spatially encodes hyperspectral images, then compresses and collects them, and then restores the original target hyperspectral information through reconstruction algorithms. This has great advantages in the field of imaging and can achieve data acquisition at a reduced cost for high-resolution cameras. However, the poor quality of hyperspectral image restoration in existing hyperspectral image reconstruction methods is a major problem that needs to be solved urgently. Summary of the invention
[0003] Based on this, it is necessary to propose a hyperspectral image reconstruction method, device, equipment and medium based on encoding and decoding to address the above problems.
[0004] To achieve the above-mentioned object, the present application provides a first aspect of a hyperspectral image reconstruction method based on coding and decoding, the method comprising:
[0005] Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0006] Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value, including:
[0007] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0008] Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image;
[0009] Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0010] In some implementations, the upsampling convolution of the compressed measurement values based on the initial reconstruction network to determine the initial estimated hyperspectral image specifically includes:
[0011] The 33×33×1 compressed measurement value is input into the upsampling convolution layer for upsampling, and an initial estimated hyperspectral image of 33×33×31 is output. The upsampling convolution layer has a 33×1×31 size. 2 convolution kernel.
[0012] In some implementations, the deep reconstruction network is based on the initially estimated hyperspectral image and the compressed measurement value, and deep feature extraction and noise removal are performed to determine the reconstructed hyperspectral image, specifically including:
[0013] Performing deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine spatial domain features;
[0014] Denoising the spatial domain features to determine the reconstructed hyperspectral image.
[0015] In some implementations, performing deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine the spatial domain features specifically includes:
[0016] Extracting compressed domain features from the initially estimated hyperspectral image through a downsampling convolution layer;
[0017] Performing feature fusion on the compression domain feature and the compression measurement value to determine a compression domain fusion feature;
[0018] The compressed domain fusion features are converted from the compressed domain to the spatial domain through an upsampling convolution layer to determine the spatial domain features.
[0019] In some implementations, removing noise from the spatial domain features to determine the reconstructed hyperspectral image specifically includes:
[0020] Splicing the learned encoding template Φ and the spatial domain feature in a third dimension to determine a splicing feature;
[0021] The splicing features are input into a Transformer network to remove noise, and the reconstructed hyperspectral image is determined.
[0022] In some embodiments, the method further comprises:
[0023] The negative lower bound of the boundary likelihood is used as the first loss function of the hyperspectral image reconstruction network, and the hyperspectral image reconstruction network is optimized according to the first loss function. The first loss function is , where p is the posterior probability distribution, q is the variational form of the posterior probability distribution, is the divergence, f rec To reconstruct hyperspectral images, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0024] In some embodiments, , where i is the i-th image pixel, d is the total number of image block pixels, const is a constant, Φ is the network weight of the encoding network, λ is the weight parameter, 1×10 -2 .
[0025] In some embodiments, the method further comprises:
[0026] according to Determine a second loss function, and optimize the hyperspectral image reconstruction network according to the second loss function, wherein: is the divergence loss function, λ is the weight parameter, 1×10 -2 , is the mean square error loss function, f rec To reconstruct hyperspectral images, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0027] To achieve the above-mentioned purpose, the second aspect of the present application provides a hyperspectral image reconstruction device based on coding and decoding, the device comprising:
[0028] A network construction module, used to construct a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0029] The encoding unit is used to downsample and convolve the original hyperspectral image based on the encoding network to determine the compressed measurement value, specifically including:
[0030] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0031] An initial reconstruction unit, configured to perform upsampling convolution on the compressed measurement value based on an initial reconstruction network to determine an initial estimated hyperspectral image;
[0032] The deep reconstruction unit is used to perform deep feature extraction and noise removal based on the deep reconstruction network, according to the initially estimated hyperspectral image and the compressed measurement value, and determine to reconstruct the hyperspectral image.
[0033] To achieve the above-mentioned object, the third aspect of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following steps:
[0034] Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0035] Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value;
[0036] Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image;
[0037] Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0038] To achieve the above-mentioned purpose, the fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:
[0039] Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0040] Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value;
[0041] Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image;
[0042] Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0043] The embodiments of the present invention have the following beneficial effects:
[0044] The present invention constructs a hyperspectral image reconstruction network under a deep learning method, and the hyperspectral image reconstruction network includes a coding network, an initial reconstruction network, and a deep reconstruction network; firstly, based on the coding network, the compression coding process is simulated, and the original hyperspectral image is downsampled and convolved to determine the compressed measurement value; secondly, based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine the initial estimated hyperspectral image; then, based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initial estimated hyperspectral image and the compressed measurement value to determine the reconstructed hyperspectral image, and finally the deep network loss function is set as the variational lower bound of the boundary likelihood, so as to obtain the reconstructed hyperspectral image through network training, and the data advantage is used to improve the quality of hyperspectral image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0046] in:
[0047] Figure 1 A schematic diagram of a process of a hyperspectral image reconstruction method based on encoding and decoding in one embodiment;
[0048] Figure 2 A schematic diagram of a compression acquisition and reconstruction process of a hyperspectral image reconstruction method based on codec in one embodiment;
[0049] Figure 3 A schematic diagram of a hyperspectral image reconstruction network in a hyperspectral image reconstruction method based on encoding and decoding in one embodiment;
[0050] Figure 4 A comparison diagram of the effects of the hyperspectral image reconstruction method based on encoding and decoding of the present invention and other hyperspectral image reconstruction methods in one embodiment;
[0051] Figure 5 is a schematic diagram of the structure of a computer device in one embodiment;
[0052] Figure 6 Schematic diagram of the structure of a computer-readable storage medium in one embodiment. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In the embodiment of the present application, a hyperspectral image reconstruction method based on codec is provided. Figure 1 , Figure 1 1 is a flow chart of a method for reconstructing a hyperspectral image based on codec in an embodiment. The method for reconstructing a hyperspectral image based on codec includes steps S1 to S4.
[0055] Step S1, constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0056] Specifically, see Figure 2 and Figure 3 , Figure 2 In the hyperspectral image compression acquisition and reconstruction process of the present invention, the target scene is converged by the objective lens, randomly encoded, and then converged to the dispersion element through a relay lens for spectral dispersion. The compressed measurement values are collected by the detector, and then the target hyperspectral image is reconstructed through the reconstruction network proposed by the present invention. Figure 3 The present invention is a schematic diagram of a hyperspectral image reconstruction network in a hyperspectral image reconstruction method based on codec in an embodiment. By constructing a hyperspectral image reconstruction network including an encoding network, an initial reconstruction network and a deep reconstruction network, compression, initial reconstruction and deep reconstruction of the original hyperspectral image are achieved, thereby obtaining a high-quality reconstructed hyperspectral image.
[0057] Step S2, based on the encoding network, downsamples and convolves the original hyperspectral image to determine the compressed measurement value, specifically including:
[0058] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0059] Specifically, first, the original hyperspectral image of size 33×33×31 is input into the downsampling convolution layer for processing. This downsampling convolution layer has 31 convolution kernels of size 33×33×1. Each convolution kernel performs a convolution operation on the spectral dimension of the original hyperspectral image to extract the feature information on that dimension. Then, after being processed by the downsampling convolution layer, the output result is a compressed measurement value of size 33×33×1. This compressed measurement value retains the spatial structure information of the original hyperspectral image, but compresses the spectral dimension into a single channel. This compressed measurement value will be used as the input of the subsequent reconstruction step. The network weights of the encoding network, that is, the learned encoding template Φ, are obtained through the training process. During the training process, the network automatically adjusts these weights according to the input original hyperspectral image and the corresponding label (usually the original image itself or a high-quality reconstructed image) to optimize the quality of the reconstructed image. The role of the encoding template Φ is to map the original hyperspectral image to the space of compressed measurements, thereby achieving image compression. The encoding process can be expressed as follows:
[0060] ,in, H down is the encoding network, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0061] By downsampling convolution and compressing the measured values, the data volume of the hyperspectral image can be effectively reduced, and the storage and transmission costs can be reduced. At the same time, since the network weights of the encoding network are used as the learned encoding template Φ, this method can retain the main structure and feature information of the original image, providing a good foundation for the subsequent reconstruction steps.
[0062] Step S3, based on the initial reconstruction network, upsampling and convolving the compressed measurement value to determine an initial estimated hyperspectral image;
[0063] Specifically, in the codec-based hyperspectral image reconstruction method, the role of the initial reconstruction network is to upsample the compressed measurement values to restore the initially estimated hyperspectral image. This process achieves data amplification by increasing the spectral dimension of the image.
[0064] In some implementations, the upsampling convolution of the compressed measurement values based on the initial reconstruction network to determine the initial estimated hyperspectral image specifically includes:
[0065] The 33×33×1 compressed measurement value is input into the upsampling convolution layer for upsampling, and an initial estimated hyperspectral image of 33×33×31 is output. The upsampling convolution layer has a 33×1×31 size. 2 convolution kernel.
[0066] First, the compressed measurements of size 33×33×1 are input to the upsampling convolutional layer for processing. This upsampling convolutional layer has 33 of size 1×1×31. 2 These convolution kernels perform convolution operations on the input compressed measurements to reconstruct the information of 31 spectral bands at each spatial location.
[0067] After the upsampling convolution layer is processed, the output result is an initial estimated hyperspectral image of size 33×33×31. This image is the same as the original image in spatial dimension, and the information of 31 bands is also restored in the spectral dimension. The initial reconstruction process can be expressed as follows:
[0068] ,in, f 1 To initially reconstruct the hyperspectral image, H up Initial reconstruction network.
[0069] Although the initially estimated hyperspectral image may contain some errors and noise, it has provided an important foundation for the subsequent deep reconstruction network. Through further processing by the deep reconstruction network, the quality and accuracy of the reconstructed image can be further improved.
[0070] Step S4, based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0071] In some implementations, the deep reconstruction network is based on the initially estimated hyperspectral image and the compressed measurement value, and deep feature extraction and noise removal are performed to determine the reconstructed hyperspectral image, specifically including:
[0072] Performing deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine spatial domain features;
[0073] Denoising the spatial domain features to determine the reconstructed hyperspectral image.
[0074] In some implementations, performing deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine the spatial domain features specifically includes:
[0075] Extracting compressed domain features from the initially estimated hyperspectral image through a downsampling convolution layer;
[0076] Specifically, the initial reconstructed hyperspectral image f 1 After the downsampling convolution layer, the compressed domain feature extraction is completed. The feature extraction process can be expressed as follows:
[0077] y 1 = F down ( f 1 ),in, y 1 is the compression domain feature, F down is the downsampling convolution operation.
[0078] Performing feature fusion on the compression domain feature and the compression measurement value to determine a compression domain fusion feature;
[0079] Specifically, the compressed measurement value obtained through the encoding network y 0 and y 1 Add and perform compression domain feature fusion. The feature fusion of this step can be expressed as: y = y 0 + y 1。
[0080] The compressed domain fusion features are converted from the compressed domain to the spatial domain through an upsampling convolution layer to determine the spatial domain features.
[0081] Specifically, the compressed domain fusion features y The upsampling convolution layer completes the conversion from compressed domain features to spatial domain. The conversion process can be expressed as:
[0082] f 2 = F up ( y ),in, f 2 is the spatial domain feature, F up is the upsampling convolution operation.
[0083] In some implementations, removing noise from the spatial domain features to determine the reconstructed hyperspectral image specifically includes:
[0084] Splicing the learned encoding template Φ and the spatial domain feature in a third dimension to determine a splicing feature;
[0085] The splicing features are input into a Transformer network to remove noise, and the reconstructed hyperspectral image is determined.
[0086] Specifically, for the spatial domain features f 2 Perform noise removal to obtain the final reconstruction result f rec , the process can be expressed as:
[0087] f rec= F denoise ( f 2 ),in, f rec To reconstruct the results, F denoise For denoising operation.
[0088] More specifically, the learned encoding template Φ and spatial domain features f 2 Perform third-dimensional stitching to obtain stitching features f con , the stitching features f con The input is into the U-net network composed of Transformer for noise removal.
[0089] In some embodiments, the method further comprises:
[0090] The negative lower bound of the boundary likelihood is used as the first loss function of the hyperspectral image reconstruction network, and the hyperspectral image reconstruction network is optimized according to the first loss function. The first loss function is , where p is the posterior probability distribution, q is the variational form of the posterior probability distribution, is the divergence, f rec To reconstruct the hyperspectral image, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0091] In some embodiments, , where i is the i-th image pixel, d is the total number of image block pixels, const is a constant, Φ is the network weight of the encoding network, and λ is the weight parameter. is 1×10-2.
[0092] Specifically, here the first likelihood term in the first loss function mentioned above is Further restrictions are made to facilitate calculation.
[0093] In some embodiments, the method further comprises:
[0094] according to Determine a second loss function, and optimize the hyperspectral image reconstruction network according to the second loss function, wherein: is the divergence loss function, λ is the weight parameter, is 1×10-2, is the mean square error loss function, f rec To reconstruct the hyperspectral image, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0095] Specifically, this is the second way to set the loss function, which sets the deep network loss function as the variational lower bound of the boundary likelihood, so as to obtain the reconstructed hyperspectral image through network training.
[0096] In some embodiments, the hyperspectral image reconstruction network can also be jointly optimized according to the first loss function and the second loss function, wherein the first loss function is used to ensure that the reconstructed hyperspectral image is as close as possible to the original image in statistical characteristics, and by minimizing the negative lower bound of the boundary likelihood, the reconstructed image can be closer to the original image in probability distribution; the second loss function combines the divergence loss and the mean square error loss, the divergence loss is used to ensure that the reconstructed image is similar to the original image in distribution, and the mean square error loss is used to ensure that the reconstructed image is as close as possible to the original image at the pixel level.
[0097] In a specific embodiment, see Figure 4 In one embodiment shown, a comparison diagram of the effects of the hyperspectral image reconstruction method based on codec of the present invention and other hyperspectral image reconstruction methods is shown. This is a comparison of the reconstruction results of the hyperspectral image reconstruction method based on codec of the present invention in bands such as 500nm, 600nm, and 700nm and the reconstruction results of other algorithms. (a) is the original image, (b)-(g) are other comparison algorithms, and (h) is the algorithm of the present invention. It is obvious that the reconstruction result of the present invention has higher reconstruction quality.
[0098] The hyperspectral image reconstruction method based on encoding and decoding proposed by the present invention combines the advantages of the encoding network, the initial reconstruction network, the Transformer network in the deep reconstruction network and the two loss functions. It can improve the quality of hyperspectral image reconstruction by combining the variational Bayesian loss function under the deep learning method, thereby taking advantage of the data advantage. Through deep feature extraction and noise removal technology, the quality and accuracy of the reconstructed image can be further improved. Among them, the encoding network can effectively compress the hyperspectral image and reconstruct it with high quality based on the compressed measurement value. This helps to save resources during storage and transmission. The deep reconstruction network can extract the deep features of the image through multi-layer convolution and feature fusion, which helps to better restore the details and texture of the image. Using the Transformer network for noise removal can effectively remove the noise generated during the reconstruction process and improve the quality of the reconstructed image. By defining the loss function, the entire network can be optimized to improve the reconstruction performance, and the method can be applied to hyperspectral images of different sizes and resolutions, with strong versatility, low cost, and high quality of hyperspectral image restoration.
[0099] In an embodiment of the present application, a hyperspectral image reconstruction device based on codec is provided. The hyperspectral image reconstruction device based on codec includes: a network construction module, a coding unit, an initial reconstruction unit and a depth reconstruction unit.
[0100] A network construction module is configured to construct a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0101] The encoding unit is configured to perform downsampling convolution on the original hyperspectral image based on the encoding network to determine the compressed measurement value, specifically including:
[0102] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0103] An initial reconstruction unit is configured to perform upsampling convolution on the compressed measurement value based on an initial reconstruction network to determine an initial estimated hyperspectral image;
[0104] The deep reconstruction unit is configured to perform deep feature extraction and noise removal based on the deep reconstruction network, according to the initially estimated hyperspectral image and the compressed measurement value, and determine to reconstruct the hyperspectral image.
[0105] In some embodiments, the initial reconstruction unit is further configured to input the 33×33×1 compressed measurement value into an upsampling convolution layer for upsampling, and output an initial estimated hyperspectral image of 33×33×31, wherein the upsampling convolution layer has a 33×33×31 size. 2 convolution kernel.
[0106] In some embodiments, the deep reconstruction unit is further configured to perform deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine spatial domain features; and perform noise removal on the spatial domain features to determine the reconstructed hyperspectral image.
[0107] In some embodiments, the deep reconstruction unit includes a feature space information supplement module and a space-spectral Transformer denoising module, and the feature space information supplement module performs deep feature extraction based on the initially estimated hyperspectral image and the compressed measurement value to determine the spatial domain features, specifically including:
[0108] Extracting compressed domain features from the initially estimated hyperspectral image through a downsampling convolution layer;
[0109] Performing feature fusion on the compression domain feature and the compression measurement value to determine a compression domain fusion feature;
[0110] The compressed domain fusion features are converted from the compressed domain to the spatial domain through an upsampling convolution layer to determine the spatial domain features.
[0111] In some implementations, the spatial-spectral Transformer denoising module removes noise from the spatial domain features to determine the reconstructed hyperspectral image, specifically comprising:
[0112] Splicing the learned encoding template Φ and the spatial domain feature in a third dimension to determine a splicing feature;
[0113] The splicing features are input into a Transformer network to remove noise, and the reconstructed hyperspectral image is determined.
[0114] In some embodiments, the spatial-spectral Transformer denoising module is composed of a Transformer network guided by a learned encoding template, the Transformer is composed of an encoder and a decoder, and each layer contains multiple self-attention mechanism modules and feedforward neural network modules for capturing long-distance dependencies in space and spectrum.
[0115] In some embodiments, the codec-based hyperspectral image reconstruction device further includes a loss function optimization unit, wherein the loss function optimization unit is configured to use the negative lower bound of the boundary likelihood as a first loss function of the hyperspectral image reconstruction network, and optimize the hyperspectral image reconstruction network according to the first loss function, wherein the first loss function is , where p is the posterior probability distribution, q is the variational form of the posterior probability distribution, is the divergence, f rec To reconstruct the hyperspectral image, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0116] In some embodiments, the loss function optimization unit is further configured to set , where i is the i-th image pixel, d is the total number of image block pixels, const is a constant, Φ is the network weight of the encoding network, and λ is the weight parameter. is 1×10-2.
[0117] In some embodiments, the loss function optimization unit is further configured to Determine a second loss function, and optimize the hyperspectral image reconstruction network according to the second loss function, wherein: is the divergence loss function, λ is the weight parameter, is 1×10-2, is the mean square error loss function, f rec To reconstruct the hyperspectral image, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
[0118] For other details about how each module in the codec-based hyperspectral image reconstruction device implements the above technical solution, please refer to the description of the codec-based hyperspectral image reconstruction method provided above, which will not be repeated here.
[0119] In an embodiment of the present application, a computer device is provided. Figure 5 , Figure 5 3 is a schematic diagram of the structure of a computer device in an embodiment. The device includes a memory 301 and a processor 302. The memory 301 stores a computer program. When the computer program is executed by the processor 302, the processor 302 performs the following steps:
[0120] Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0121] Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value, including:
[0122] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0123] Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image;
[0124] Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0125] Among them, the processor 302 can also be called a CPU (Central Processing Unit), and the processor 302 may be an integrated circuit chip with signal processing capabilities; the processor 302 can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor 302 can also be any conventional processor, etc.
[0126] In an embodiment of the present application, a computer readable storage medium is provided. Figure 6 , Figure 6 The structure diagram of a computer-readable storage medium in an embodiment is a schematic diagram, on which a readable computer program 401 is stored; wherein the computer program 401 may be stored in the above storage medium in the form of a software product, and includes a number of instructions for causing a computer device (which may be a personal computer, a server machine, or a network device, etc.) or a processor to perform the following steps:
[0127] Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network;
[0128] Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value, including:
[0129] The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight of the encoding network is the learned encoding template Φ;
[0130] Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image;
[0131] Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement value to determine a reconstructed hyperspectral image.
[0132] The aforementioned storage media include: U disk, mobile hard disk, magnetic disk or CD, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program code, or terminal devices such as computers, service machines, mobile phones, and tablets.
[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0134] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0135] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A hyperspectral image reconstruction method based on encoding and decoding, characterized in that: The method includes: Constructing a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network; Based on the encoding network, the original hyperspectral image is downsampled and convolved to determine the compressed measurement value, including: The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight Φ of the encoding network is the learned encoding template; Based on the initial reconstruction network, the compressed measurement value is upsampled and convolved to determine an initial estimated hyperspectral image; Based on the deep reconstruction network, deep feature extraction and noise removal are performed according to the initially estimated hyperspectral image and the compressed measurement values to determine the reconstructed hyperspectral image, including performing deep feature extraction according to the initially estimated hyperspectral image and the compressed measurement values to determine spatial domain features, performing noise removal on the spatial domain features to determine the reconstructed hyperspectral image; the performing deep feature extraction according to the initially estimated hyperspectral image and the compressed measurement values to determine the spatial domain features specifically includes: extracting compressed domain features from the initially estimated hyperspectral image through a downsampling convolution layer, performing feature fusion on the compressed domain features and the compressed measurement values to determine compressed domain fusion features, and converting the compressed domain fusion features from the compressed domain to the spatial domain through an upsampling convolution layer to determine the spatial domain features; the performing noise removal on the spatial domain features to determine the reconstructed hyperspectral image specifically includes: performing third-dimensional splicing on the learned encoding template and the spatial domain features to determine splicing features, inputting the splicing features into a Transformer network to perform noise removal, and determining the reconstructed hyperspectral image.
2. The method for reconstructing hyperspectral images based on coding and decoding according to claim 1, characterized in that: The method of performing upsampling convolution on the compressed measurement value based on the initial reconstruction network to determine the initial estimated hyperspectral image specifically includes: The 33×33×1 compressed measurement value is input into the upsampling convolution layer for upsampling, and an initial estimated hyperspectral image of 33×33×31 is output. The upsampling convolution layer has a 33×1×31 size. 2 convolution kernel.
3. The method for reconstructing hyperspectral images based on coding and decoding according to claim 2, characterized in that: The method further comprises: The negative lower bound of the boundary likelihood is used as the first loss function of the hyperspectral image reconstruction network, and the hyperspectral image reconstruction network is optimized according to the first loss function. The first loss function is , where p is the posterior probability distribution, q is the variational form of the posterior probability distribution, is the divergence, f rec To reconstruct hyperspectral images, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
4. The method for reconstructing hyperspectral images based on coding and decoding according to claim 3, characterized in that: , where i is the i-th image pixel, d is the total number of image block pixels, const is a constant, Φ is the network weight of the encoding network, λ is the weight parameter, 1×10 -2 .
5. The method for reconstructing hyperspectral images based on coding and decoding according to claim 4, characterized in that: The method further comprises: according to Determine a second loss function, and optimize the hyperspectral image reconstruction network according to the second loss function, wherein: is the divergence loss function, λ is the weight parameter, 1×10 -2 , is the mean square error loss function, f rec To reconstruct hyperspectral images, y 0 is the compression measurement value, f 0 is the original hyperspectral image.
6. A hyperspectral image reconstruction device based on codec, characterized in that: The device includes: A network construction module, used to construct a hyperspectral image reconstruction network, wherein the hyperspectral image reconstruction network includes an encoding network, an initial reconstruction network, and a deep reconstruction network; The encoding unit is used to downsample and convolve the original hyperspectral image based on the encoding network to determine the compressed measurement value, specifically including: The original hyperspectral image of 33×33×31 is input into the downsampling convolution layer for downsampling, and a compressed measurement value of 33×33×1 is output. The downsampling convolution layer has 31 convolution kernels of size 33×33×1, and the network weight Φ of the encoding network is the learned encoding template; An initial reconstruction unit, configured to perform upsampling convolution on the compressed measurement value based on an initial reconstruction network to determine an initial estimated hyperspectral image; A deep reconstruction unit is used to perform deep feature extraction and noise removal based on the initial estimated hyperspectral image and the compressed measurement value based on the deep reconstruction network to determine the reconstructed hyperspectral image, including performing deep feature extraction based on the initial estimated hyperspectral image and the compressed measurement value to determine the spatial domain feature, performing noise removal on the spatial domain feature, and determining the reconstructed hyperspectral image; the deep feature extraction based on the initial estimated hyperspectral image and the compressed measurement value to determine the spatial domain feature specifically includes: extracting compressed domain features from the initial estimated hyperspectral image through a downsampling convolution layer, performing feature fusion on the compressed domain features and the compressed measurement value to determine the compressed domain fusion feature, and converting the compressed domain fusion feature from the compressed domain to the spatial domain through an upsampling convolution layer to determine the spatial domain feature; the noise removal on the spatial domain feature to determine the reconstructed hyperspectral image specifically includes: splicing the learned encoding template and the spatial domain feature in the third dimension to determine the splicing feature, inputting the splicing feature into a Transformer network to perform noise removal, and determining the reconstructed hyperspectral image.
Citation Information
Patent Citations
Hyperspectral image super-resolution reconstruction method and device and electronic equipment
CN113139902A
Hyperspectral image super-resolution reconstruction method based on two-stage strategy
CN113781306A
