Hyperspectral image reconstruction system based on binary neural network

Abstract

The invention discloses a hyperspectral image reconstruction system based on a binary neural network, and the system comprises a binary neural network model used for reconstructing a hyperspectral image, and the binary neural network model comprises a degeneration parameter estimator which is used for estimating degeneration degree information based on a measurement value, and outputting K groups of degeneration parameters; k is a positive integer; the initialization extractor is used for extracting full-precision information from the measured value to obtain an initial hyperspectral reconstruction signal; and the K iterative reconstruction modules are connected to the output ends of the degradation parameter estimator and the initialization extractor and are used for reconstructing a hyperspectral image according to the K groups of degradation parameters and the initial hyperspectral reconstruction signal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a hyperspectral image reconstruction system based on a binary neural network. Background Art

[0002] Hyperspectral images (HSIs) are known for their broad spectral bands and can capture richer and more detailed information. Due to this characteristic, hyperspectral images have been widely used in fields such as target tracking, remote sensing, and medical image analysis. To efficiently acquire hyperspectral images, many researchers have adopted the coded aperture snapshot spectral imaging (CASSI) system. This system uses a coded aperture (i.e., a physical mask) to encode the 3D HSI cube and a disperser to separate the spectral bands, and these signals are then integrated into a detector array to generate a 2D compressed measurement map. Hyperspectral image reconstruction aims to solve its inverse problem: reconstruct the original 3D HSI cube from the compressed 2D compressed measurement map.

[0003] Existing methods are mainly based on deep learning models such as convolutional neural networks (CNNs). However, these methods require high computational costs and large amounts of memory storage and rely on high-performance hardware such as advanced graphics processing units (GPUs), which are unaffordable for edge devices.

[0004] Recently, methods based on binary neural networks (BNNs) have been introduced to improve the efficiency of hyperspectral imaging (SCI) reconstruction. However, these methods cannot capture long-range dependencies. In addition, these methods are mainly end-to-end and usually directly learn a rough mapping from two-dimensional measurements to three-dimensional hyperspectral images, resulting in a lack of theoretical basis and interpretability for the algorithms, and these drawbacks limit the reconstruction performance. Summary of the Invention

[0005] Aiming at the defects existing in the above-mentioned prior art, the present invention provides a hyperspectral image reconstruction system based on a binary neural network, which can reduce computational and memory costs and efficiently reconstruct high-quality hyperspectral images.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A hyperspectral image reconstruction system based on a binary neural network, comprising a binary neural network model for reconstructing hyperspectral images. The binary neural network model includes: a degradation parameter estimator for estimating degradation degree information based on measurement values and outputting K sets of degradation parameters, where K is a positive integer; an initialization extractor for extracting full-precision information from the measurement values to obtain an initial hyperspectral reconstruction signal; and K iterative reconstruction modules connected to the output ends of the degradation parameter estimator and the initialization extractor for reconstructing a hyperspectral image according to the K sets of degradation parameters and the initial hyperspectral reconstruction signal.

[0008] Further, the degradation parameter estimator takes the measurement values as input, captures key clues from the measurement values, estimates the degradation degree information, and outputs the K sets of degradation parameters to provide prior information for reconstruction.

[0009] Further, among the K iterative reconstruction modules, the input end of the first iterative reconstruction module is connected to the output end of the initialization extractor, the input ends of the subsequent K - 1 iterative reconstruction modules are respectively connected to the output end of the previous iterative reconstruction module, and at the same time, the input ends of the K iterative reconstruction modules are all connected to the output end of the degradation parameter estimator.

[0010] Further, the degradation parameter estimator includes: a convolutional layer, two binary convolutional blocks connected after the convolutional layer, a global average pooling layer connected after the two binary convolutional blocks, and a Softplus activation layer connected after the global average pooling layer.

[0011] Further, the binary convolutional block includes: a spectral feature adjustment layer taking a full-precision hyperspectral signal as input and adjusting the density of the full-precision hyperspectral signal on each channel to obtain an adjusted signal; an RSign activation layer connected to the output end of the spectral feature adjustment layer for binarizing the adjusted signal into a 1-bit activation signal; a Sign activation layer taking a full-precision convolutional kernel weight as input and converting the full-precision convolutional kernel weight into a 1-bit weight; a binary convolutional layer connected to the output ends of the Sign activation layer and the RSign activation layer for performing a binarization operation on the 1-bit activation signal and the 1-bit weight; a multiplier for multiplying the output of the binary convolutional layer by the absolute average value of the full-precision convolutional kernel weight; a first RPReLU activation layer connected to the output end of the multiplier; an adder for adding the output of the first RPReLU activation layer to the full-precision hyperspectral signal; and a second RPReLU activation layer taking the output of the adder as input to obtain the output of the binary convolutional block.

[0012] Further, the iterative reconstruction module includes a linear solver and a Gaussian denoiser; wherein, the linear solver of the first iterative reconstruction module takes the output of the initialization extractor as input, and the linear solvers of the subsequent K-1 iterative reconstruction modules respectively take the output of the previous iterative reconstruction module as input; the input of the Gaussian denoiser includes the output of the linear solver and one set of the K sets of degradation parameters.

[0013] Further, the formula of the linear solver is:

[0014]

[0015] wherein, x k+1 represents the output of the linear solver, k = 0, 1, 2, …, K-1 represents the iteration index; when k = 0, z k is the initial hyperspectral reconstruction signal output by the initialization extractor, and when k ≥ 1, z k is the output of the k-th iterative reconstruction module; Φ is the sensing matrix of the system, y is the measurement value; the symbol represents the Hadamard division of element-wise division, and Diag is the operation of extracting diagonal elements; μ k+1 represents the parameter value output by the degradation parameter estimator.

[0016] Further, the Gaussian denoiser adopts a three-level U-shaped architecture, which sequentially includes from input to output: an embedding layer, a first binary convolutional block, a first binary residual Mamba module, a first downsampling layer, a second binary residual Mamba module, a second downsampling layer, a third binary residual Mamba module, a first upsampling layer, a first binary channel reduction block, a fourth binary residual Mamba module, a second upsampling layer, a second binary channel reduction block, a fifth binary residual Mamba module, a second binary convolutional block, and a mapping layer; wherein, the input of the embedding layer includes the hyperspectral signal x k+1 output by the linear solver. The output of the embedding layer, in addition to being sent to the first binary convolutional block, is also added to the output of the second binary convolutional block and then sent to the mapping layer; the output of the first binary residual Mamba module, in addition to being sent to the first downsampling layer, is also sent to the second binary channel reduction block; the output of the second binary residual Mamba module, in addition to being sent to the second downsampling layer, is also sent to the first binary channel reduction block; the denoised hyperspectral signal z k+1 output by the mapping layer is the output of the Gaussian denoiser.

[0017] Further, the first to fifth binary residual Mamba modules have the same structure, all including: a first LayerNorm layer, a binary state space model, a second LayerNorm layer, a first binary channel increase block, a third binary convolution block, and a third binary channel reduction block, which are connected in sequence from the input to the output; wherein, the input of the second LayerNorm layer is the sum of the output of the binary state space model and the input of the first LayerNorm layer, and the sum of the input of the second LayerNorm layer and the output of the third binary channel reduction block is used as the final output of the binary residual Mamba module.

[0018] Further, the input of the binary state space model passes through two branches: the first branch includes a second binary channel increase block, a fourth binary convolution block, a two-dimensional selective scanning module, and a third LayerNorm layer connected in sequence, and the second branch includes a third binary channel increase block; at the output ends of the two branches, the features output by the two branches are aggregated through a Hadamard product, and then projected through a fourth binary channel reduction block to obtain an output with the same shape as the input.

[0019] The beneficial effects of the technical solution of the present invention are at least reflected in: the hyperspectral image reconstruction system based on the binary neural network provided by the present invention, the binary neural network model included therein captures key clues (such as degradation parameters) from the measured values (images to be processed) through a degradation parameter estimator, estimates the degradation degree information, efficiently provides prior information for reconstruction, improves the interpretability of the hyperspectral image reconstruction process, and improves the spectral image reconstruction performance.

[0020] In addition, the output parameters of the degradation parameter estimator will act on the Gaussian denoiser, affecting the effect of each iteration, and the binary state space model (BiSSM) included in the Gaussian denoiser models the global spectral interaction through the binary state space model (BiSSM), establishes the non-linear relationship between the bands of the hyperspectral image, can capture long-range dependencies, greatly reduces the complexity, reduces the calculation and memory requirements, and has significant advantages in terms of information integration ability, robustness, and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a schematic flowchart of the hyperspectral image reconstruction method based on the binary neural network according to the embodiment of the present invention;

[0022] Figure 2 is a schematic structural diagram of the binary neural network model in the hyperspectral image reconstruction system based on the binary neural network according to the embodiment of the present invention;

[0023] Figure 3 is a schematic structural diagram of the degradation parameter estimator proposed by the embodiment of the present invention;

[0024] Figure 4 This is a schematic structural diagram of the Binary Convolutional Block (BCB) proposed in an embodiment of the present invention;

[0025] Figure 5 This is a schematic structural diagram of the Gaussian denoiser proposed in an embodiment of the present invention;

[0026] Figure 6 This is a schematic structural diagram of the Binary Residual Mamba Block (RBMB) proposed in an embodiment of the present invention;

[0027] Figure 7 This is a schematic structural diagram of the Binary State Space Model (BiSSM) proposed in an embodiment of the present invention;

[0028] Figure 8 This is a schematic structural diagram of the electronic device proposed in an embodiment of the present invention. Detailed implementation manners

[0029] The present invention will be further described below in conjunction with the accompanying drawings, specific implementation manners, and embodiments. The purpose of providing the embodiments is only for illustration and not for any limitation.

[0030] An embodiment of the present invention first proposes a hyperspectral image reconstruction system for a binary neural network. The system includes a binary neural network model for reconstructing hyperspectral images, and the architecture of the model is as Figure 2 shown; the system may further include a preprocessing module, a training module, and a prediction module. The preprocessing module is used to obtain a hyperspectral image dataset, perform image preprocessing, and divide the preprocessed dataset into a training set and a test set; the training module is used to train the binary neural network model based on the training set to obtain a binary neural network model with optimal parameters; the prediction module is used to input the test set into the binary neural network model with optimal parameters to reconstruct the hyperspectral image.

[0031] The binary neural network model of the present invention takes the measurement values obtained based on the sampling principle of a single-dispersion-element coded aperture snapshot spectral imaging system (SD-CASSI) as input and outputs the reconstructed hyperspectral image. Please refer to Figure 2, in some specific embodiments, the binary neural network model includes a degradation parameter estimator, an initialization extractor, and K iterative reconstruction modules, where K is a positive integer; among them, the degradation parameter estimator and the initialization extractor are two parallel branches, both taking the measurement value (i.e., the image to be processed) as input; the K iterative reconstruction modules are connected in series in sequence, where the first iterative reconstruction module is connected after the initialization extractor, and the output of the last iterative reconstruction module is the final output of the binary neural network model (i.e., the reconstruction result); the degradation parameter estimator takes the measurement value as input, captures key clues from the measurement value, estimates the degradation degree information, and outputs K sets of degradation parameters. The K sets of degradation parameters contain 2K parameters, denoted as τ = [τ1, τ2,..., τ K , μ = [μ1, μ2,..., μ K . The K sets of degradation parameters [τ j , μ j (j = 1, 2,..., K) output by the degradation parameter estimator are respectively input to the K iterative reconstruction modules, efficiently providing prior information for reconstruction and improving the interpretability of the reconstruction method. The initialization extractor can be, for example, a 1×1 convolutional layer, which takes the measurement value as input and extracts full-precision spectral characteristic information from the measurement value to obtain the initial hyperspectral reconstruction signal z0. The first iterative reconstruction module takes the initial hyperspectral reconstruction signal z0 and the degradation parameters [τ1, μ1] as input, and the second to the Kth iterative reconstruction modules respectively take the output of the previous iterative reconstruction module as input, and at the same time respectively receive the degradation parameters [τ2, μ2], …, [τ K , μ K as input prior information. Through continuous iterative reconstruction, the reconstructed hyperspectral image is finally output through the Kth iterative reconstruction module.

[0032] Please refer to Figure 3 , in some specific embodiments, the degradation parameter estimator includes a 1×1 convolutional layer, two binary convolutional blocks (BCB), a global average pooling layer, and a Softplus activation layer connected in series in sequence. Among them, the Softplus activation layer uses the Softplus function as the activation function to ensure that the prior parameter values output by the degradation parameter estimator are greater than zero.

[0033] Please refer to Figure 4 , in some specific embodiments, the binary convolutional block (BCB) includes a spectral feature adjustment layer, a Sign activation layer, a RSign activation layer, a binary convolutional layer, and two RPReLU activation layers.

[0034] In the original binary neural network (BNN), the full-precision hyperspectral signal x r is binarized into the binary activation value x by the Sign functionb , specifically as follows:

[0035]

[0036] Among them, the subscripts b and r represent binary values and real values respectively.

[0037] However, due to the limitation of specific wavelengths, hyperspectral signals exhibit different densities and distributions in the spectral dimension. Therefore, in the binary convolutional block (BCB) of the embodiments of the present invention, the full-precision hyperspectral signal x is first adaptively adjusted through a spectral feature adjustment layer r The density on each channel, and the adjusted signal is Xa = q·Xr, where Xr is the matrix of the full-precision hyperspectral signal, and x r ∈Xr; where q is a learnable parameter; then, the adjusted signal Xa is binarized into a 1-bit activation signal X through the RSign function with a channel-level learnable threshold α b :

[0038]

[0039] Among them, i is the index of the channel, indicating that the threshold can be different values on different channels.

[0040] Continue to refer to Figure 4 , in the binary convolutional block (BCB) of the embodiments of the present invention, the Sign activation layer takes the full-precision convolutional kernel weight w r as the input and converts the full-precision convolutional kernel weight into a 1-bit weight w b , and the specific method is as follows:

[0041]

[0042] Among them, w r ∈W r , and W r is the weight matrix. By multiplying with the absolute average value of the weights , the gap between the binarized weight w b and the full-precision convolutional kernel weight w r is significantly reduced. In this way, the computationally intensive floating-point matrix multiplication operation in the full-precision convolution can be replaced by pure logic XNOR (exclusive NOR) operation and bit-counting operation Bit-Count.

[0043] Y b = X b * W b = Bit-Count(XNOR(X b , W b )), (4)

[0044] where * represents convolution. Since the value range of the full-precision hyperspectral signal Xr is greater than that of the output Yb of the binary convolutional layer, directly combining them using the identity mapping may obscure the information of Y b Therefore, first, Y b passes through the RPReLU activation layer, and the RPReLU function is used to adjust its value range to match that of the 1-bit convolutional output and enhance the non-linear expression ability of the model. The RPReLU function is as follows:

[0045]

[0046] where β i , γ i and ζ i are learnable parameters.

[0047] The binary convolutional block (BCB) of the embodiment of the present invention can propagate full-precision information through a bypass identity shortcut connection. Finally, the RPReLU activation layer is used again to obtain the output signal of the binary convolutional block (BCB), which introduces a non-linear transformation into the model, enabling the network to learn complex features and relationships.

[0048] The iterative reconstruction module of the embodiment of the present invention includes a linear solver and a Gaussian denoiser connected in series. In the binary neural network model of the present invention, the linear solver of the first iterative reconstruction module takes the output of the initialization extractor as input, and the linear solvers of the subsequent K - 1 iterative reconstruction modules take the output of the previous iterative reconstruction module as input respectively; and the input of the Gaussian denoiser includes: the output of the connected linear solver, and one of the K groups of degradation parameters.

[0049] In some specific embodiments, the formula of the linear solver is:

[0050]

[0051] where x k+1 represents the output of the linear solver, k = 0, 1, 2, …, K - 1 represents the index of iteration; when k = 0, z k is the initial hyperspectral reconstruction signal output by the initialization extractor, when k ≥ 1, z k is the output of the k-th iterative reconstruction module; Φ is the sensing matrix of the system, y is the measurement value; the symbol represents the Hadamard division of element-wise division, Diag is the operation of extracting diagonal elements; μ k+1 represents the parameter value output by the degradation parameter estimator.

[0052] Please refer to Figure 5, in some specific embodiments, the Gaussian denoiser adopts a three-level U-shaped architecture, which sequentially includes, from input to output: an embedding layer, a binary convolutional block (BCB) 11, a binary residual Mamba module (RBMB) 21, a downsampling layer 31, a binary residual Mamba module (RBMB) 22, a downsampling layer 32, a binary residual Mamba module (RBMB) 23, an upsampling layer 41, a binary channel reduction block (BCDB) 51, a binary residual Mamba module (RBMB) 24, an upsampling layer 42, a binary channel reduction block (BCDB) 52, a binary residual Mamba module (RBMB) 25, a binary convolutional block (BCB) 12, and a mapping layer; wherein, the input of the embedding layer includes the hyperspectral signal x output by the linear solver k+1 , in addition to being fed into the binary convolutional block (BCB) 11, the output of the embedding layer is also added to the output of the binary convolutional block (BCB) 12 and then fed into the mapping layer; in addition to being fed into the downsampling layer 31, the output of the binary residual Mamba module (RBMB) 21 is also fed into the binary channel reduction block (BCDB) 52; in addition to being fed into the downsampling layer 32, the output of the binary residual Mamba module (RBMB) 22 is also fed into the binary channel reduction block (BCDB) 51; the denoised hyperspectral signal z output by the mapping layer k+1 is the output of the Gaussian denoiser.

[0053] Specifically, in this Gaussian denoiser, the embedding layer and the mapping layer can be respectively implemented by a 3×3 convolutional layer; the structure and function of the binary convolutional block (BCB) have been described in the foregoing content and will not be elaborated here.

[0054] The internal structure of the binary residual Mamba module (RBMB) is as Figure 6 shown, including: a first LayerNorm layer, a binary state space model (BiSSM), a second LayerNorm layer, a binary channel increase block (BCIB), a binary convolutional block (BCB), and a binary channel reduction block (BCDB) connected in sequence from input to output; wherein, the input of the second LayerNorm layer is the sum of the output of the binary state space model (BiSSM) and the input of the first LayerNorm layer, and the sum of the input of the second LayerNorm layer and the output of the last binary channel reduction block (BCDB) is used as the final output of the binary residual Mamba module.

[0055] Continuing to refer to Figure 6 , for a binary residual Mamba module, given the input feature F in, first, the data is normalized using the first LayerNorm layer (LN), followed by a Binary State Space Model (BiSSM) to capture long-range dependencies. Then, a skip connection is used to add the input feature F in to the output of the BiSSM to obtain the intermediate feature F mid , and the formula is as follows:

[0056] F mid = BiSSM(LN(F in )) + F in (7)

[0057] Since the Structured State Space Model (SSM) processes the flattened feature map as a one-dimensional token sequence, the number of neighboring pixels in the sequence is largely affected by the flattening strategy. For example, when using a four-way flattening strategy, the anchored pixel can only obtain four nearest neighbors. That is, pixels that are spatially adjacent in the two-dimensional feature map are far apart in the flattened one-dimensional token sequence, and this excessive distance may lead to the loss of local pixel information. Therefore, an additional local convolutional layer is introduced after the Binary State Space Model (BiSSM) in the embodiments of the present invention to help restore neighborhood similarity. Specifically, first, the second LayerNorm layer (LN) is used to normalize F mid , and then a Binary Channel Increase Block (BCIB), a Binary Convolution Block (BCB), and a Binary Channel Decrease Block (BCDB) are used to learn local features, as follows:

[0058] F out = BCDB(BCB(BCIB(LN(F mid )))) + F mid . (8)

[0059] Specifically, the Binary Channel Increase Block (BCIB) copies the input LN(F mid ) into m copies, and each copy passes through an independent Binary Convolution Block (BCB), and then all the copies are concatenated. In contrast, the Binary Channel Decrease Block (BCDB) divides the input into m parts by channel, each part passes through an independent Binary Convolution Block (BCB), then the average value of each part is calculated, and finally they are concatenated to obtain the final output F out of the Binary Residual Mamba module.

[0060] Please refer to Figure 7, in some specific embodiments, the input of the Binary State Space Model (BiSSM) passes through two parallel branches: the first branch includes a Binary Channel Increment Block (BCIB) 01, a Binary Convolution Block (BCB) 02, a 2D Selective Scanning Module (2D-SSM), and a LayerNorm layer 03 connected in sequence; the second branch includes a Binary Channel Increment Block (BCIB) 04; at the output ends of the two branches, the features output by the two branches are aggregated through the Hadamard product ⊙, and then projected through the final Binary Channel Decrement Block (BCDB) 05 to obtain an output with the same shape as the input. By modeling the global spectral interaction through the Binary State Space Model (BiSSM) and establishing the non-linear relationship between the bands of the hyperspectral image, the long-range dependence relationship can be captured, the complexity can be greatly reduced, and the computational and memory requirements can be reduced. It has significant advantages in terms of information integration ability, robustness, and accuracy.

[0061] Continue to refer to Figure 7 , the 2D Selective Scanning Module (2D-SSM) projects the given sequence x(t) into a new sequence y(t) through the hidden state h(t), as shown in the following formula (8). The discrete state space equation is used to model the interaction between the tokens, indicating that the state space modeling has causal characteristics:

[0062]

[0063] y(t) = Ch(t) + Dx(t), (9)

[0064] where is the control matrix, Δ represents the step size, B is the input matrix, C is the output matrix, and D is the parameter matrix. These matrices expand the features to the hidden state dimension.

[0065] Specifically, in the 2D Selective Scanning Module (2D-SSM) of the embodiment of the present invention, the parameter selection is expressed as:

[0066]

[0067] This enables the parameters to adapt to the input, thereby enhancing the processing of selective information in the sequence. Among them, Chunk() means dividing the tensor within the brackets evenly into a specific number of tensor chunks along the specified dimension and returning a tuple with the tensor chunks as elements. B and C are obtained from a Binary Channel Reduction Block (BCDB) 06, which reduces the number of input channels to a predetermined value. Specifically, BCIB clones the input into m copies (m is a positive integer). Each copy passes through a separate BCB, and then all the outputs are concatenated by channel. In contrast, BCDB splits the input into m parts by channel (m is a positive integer), and each part passes through a separate BCB. Then, an average is taken for each part, and finally, they are concatenated by channel to obtain the output. At the same time, to improve efficiency and reduce redundant parameters, first, a Binary Channel Reduction Block (BCDB) 07 is passed through to extract information and reduce the dimension, and then a Binary Channel Increase Block (BCIB) 08 is passed through to increase the dimension and restore the original shape, finally obtaining Δ. The embedding layer 09 performs feature extraction to obtain A and D, where A is the control matrix. Finally, a two-dimensional selective scan is performed on B, C, Δ, A, and D, and the obtained output is sent to the LayerNorm layer (LN) 03.

[0068] Another embodiment of the present invention provides a hyperspectral image reconstruction method based on a binary neural network, referring to Figure 1 , the method includes:

[0069] Obtain a hyperspectral image dataset and perform preprocessing, and divide the preprocessed dataset into a training set and a test set; construct a binary neural network model for hyperspectral image reconstruction, and the model structure has been described in detail in the foregoing embodiments and will not be elaborated here;

[0070] Input the training set into the binary neural network model for training to obtain a binary neural network model with optimal parameters; the training objective is to minimize the reconstruction root mean square error (RMSE) between the reconstructed HSI and the real HSI, that is, the loss function is RMSE; the training process is the training process of a standard deep neural network, and the iteration stops when it reaches a predetermined number of times;

[0071] Input the test set into the binary neural network model with the optimal parameters to reconstruct the hyperspectral image.

[0072] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned hyperspectral image reconstruction method can be implemented. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0073] Finally, an embodiment of the present invention further provides an electronic device. Refer to Figure 8 , the electronic device includes a processor and a memory. A number of computer programs are stored on the memory for the processor to execute. When the processor executes these computer programs, the above-mentioned hyperspectral image reconstruction method based on a binary neural network can be implemented.

[0074] Optionally, the electronic device may include one or more memories. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc., and the data storage area can store relevant data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or other volatile solid-state storage devices. In some embodiments, the memory may further include a memory remotely set relative to the processor, and these remote memories can be connected to the computer terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] Optionally, the electronic device may include one or more processors, which may be a Central Processing Unit (CPU), a Microprocessor Unit (MPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices (PLDs), discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor. The processor is the control center of the computer device and connects various parts of the computer device using various interfaces and lines.

[0076] Optionally, the electronic device may also communicate with one or more external devices (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be performed through an input / output (I / O) interface. Moreover, the electronic device may also communicate with one or more networks (such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet) through a network adapter. Other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0077] Finally, it should be noted that in the description of the technical solution of the present invention, terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0078] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those skilled in the technical field to which the present invention pertains, without departing from the concept of the present invention, several equivalent substitutions or obvious variations can be made, and as long as the performance or use is the same, they should all be regarded as belonging to the protection scope of the present invention.

Claims

1. A hyperspectral image reconstruction system based on a binary neural network, characterized in that It includes a binary neural network model for reconstructing hyperspectral images, and the binary neural network model includes: A degradation parameter estimator for estimating degradation degree information based on measurement values and outputting K sets of degradation parameters; K is a positive integer; An initialization extractor for extracting full-precision information from the measurement values to obtain an initial hyperspectral reconstruction signal; and K iterative reconstruction modules connected to the output ends of the degradation parameter estimator and the initialization extractor for reconstructing a hyperspectral image according to the K sets of degradation parameters and the initial hyperspectral reconstruction signal.

2. The hyperspectral image reconstruction system according to claim 1, characterized in that, The degradation parameter estimator takes the measurement values as input, captures key clues from the measurement values, estimates the degradation degree information, and outputs the K sets of degradation parameters to provide prior information for reconstruction.

3. The hyperspectral image reconstruction system according to claim 1 or 2, characterized in that, Among the K iterative reconstruction modules, the input end of the first iterative reconstruction module is connected to the output end of the initialization extractor, the input ends of the subsequent K-1 iterative reconstruction modules are respectively connected to the output end of the previous iterative reconstruction module, and at the same time, the input ends of the K iterative reconstruction modules are all connected to the output end of the degradation parameter estimator.

4. The hyperspectral image reconstruction system according to claim 1 or 2, characterized in that, The degradation parameter estimator includes: a convolutional layer, two binary convolutional blocks connected after the convolutional layer, a global average pooling layer connected after the two binary convolutional blocks, and a Softplus activation layer connected after the global average pooling layer.

5. The hyperspectral image reconstruction system according to claim 4, wherein The binary convolutional block includes: A spectral feature adjustment layer taking a full-precision hyperspectral signal as input, adjusting the density of the full-precision hyperspectral signal on each channel to obtain an adjusted signal; An RSign activation layer connected to the output end of the spectral feature adjustment layer for binarizing the adjusted signal into a 1-bit activation signal; A Sign activation layer taking a full-precision convolutional kernel weight as input and converting the full-precision convolutional kernel weight into a 1-bit weight; A binary convolutional layer connected to the output ends of the Sign activation layer and the RSign activation layer for performing a binarization operation on the 1-bit activation signal and the 1-bit weight; A multiplier for multiplying the output of the binary convolutional layer by the absolute average value of the full-precision convolutional kernel weight; A first RPReLU activation layer connected to the output end of the multiplier; An adder for adding the output of the first RPReLU activation layer to the full-precision hyperspectral signal; A second RPReLU activation layer taking the output of the adder as input to obtain the output of the binary convolutional block.

6. The hyperspectral image reconstruction system according to any one of claims 1-5, characterized in that, The iterative reconstruction module includes a linear solver and a Gaussian denoiser; among them, the linear solver of the first iterative reconstruction module takes the output of the initialization extractor as input, and the linear solvers of the subsequent K-1 iterative reconstruction modules respectively take the output of the previous iterative reconstruction module as input; the input of the Gaussian denoiser includes the output of the linear solver and one of the K sets of degradation parameters.

7. The hyperspectral image reconstruction system according to claim 6, wherein, The formula of the linear solver is: where x k+1 represents the output of the linear solver, and k = 0, 1, 2, …, K−1 represents the index of iteration; when k = 0, z k is the initial hyperspectral reconstruction signal output by the initialization extractor, and when k ≥ 1, z k is the output of the k-th iterative reconstruction module; Φ is the sensing matrix of the system, and y is the measurement value; the symbol represents Hadamard division by element-wise division, and Diag is the operation of extracting diagonal elements; μ k+1 represents the parameter value output by the degradation parameter estimator.

8. The hyperspectral image reconstruction system according to claim 7, wherein The Gaussian denoiser adopts a three-level U-shaped architecture, which successively includes, from input to output: an embedding layer, a first binary convolutional block, a first binary residual Mamba module, a first downsampling layer, a second binary residual Mamba module, a second downsampling layer, a third binary residual Mamba module, a first upsampling layer, a first binary channel reduction block, a fourth binary residual Mamba module, a second upsampling layer, a second binary channel reduction block, a fifth binary residual Mamba module, a second binary convolutional block, and a mapping layer; wherein, the input of the embedding layer includes the hyperspectral signal x output by the linear solver k+1 , the output of the embedding layer, in addition to being fed into the first binary convolutional block, is also added to the output of the second binary convolutional block and then fed into the mapping layer; the output of the first binary residual Mamba module, in addition to being fed into the first downsampling layer, is also fed into the second binary channel reduction block; the output of the second binary residual Mamba module, in addition to being fed into the second downsampling layer, is also fed into the first binary channel reduction block; the denoised hyperspectral signal z k+1 output by the mapping layer is the output of the Gaussian denoiser.

9. The hyperspectral image reconstruction system according to claim 8, wherein The first to fifth binary residual Mamba modules have the same structure, all of which include: a first LayerNorm layer, a binary state space model, a second LayerNorm layer, a first binary channel increase block, a third binary convolutional block, and a third binary channel reduction block, which are connected in sequence from the input to the output; wherein, the input of the second LayerNorm layer is the sum of the output of the binary state space model and the input of the first LayerNorm layer, and the sum of the input of the second LayerNorm layer and the output of the third binary channel reduction block is used as the final output of the binary residual Mamba module.

10. The hyperspectral image reconstruction system according to claim 9, wherein The input of the binary state space model passes through two branches: the first branch includes a second binary channel increase block, a fourth binary convolutional block, a two-dimensional selective scan module, and a third LayerNorm layer connected in sequence, and the second branch includes a third binary channel increase block; at the output ends of the two branches, the features output by the two branches are aggregated through a Hadamard product, and then projected through a fourth binary channel reduction block to obtain an output with the same shape as the input.

Images