A compressed sensing image reconstruction method based on a gated recurrent unit

Abstract

The application belongs to the technical field of signal processing and deep learning, and particularly relates to a natural image reconstruction method applied to compressed sensing. In view of the problem of how to realize sparse sampling and high-quality reconstruction of natural images, the application proposes a novel deep unfolding network with persistent long short-term memory for compressed sensing image reconstruction. The network simultaneously realizes constraint optimization of image adaptive sampling and recovery, solves the problems of insufficient representation and internal information loss in long-distance transmission of feature information in existing deep unfolding networks by using an attention mechanism and long-term memory flow. An intra-level attention mechanism and an inter-level memory mechanism are designed and integrated in a proximal mapping module of the unfolding network, so as to realize persistent storage and transmission of long short-term feature memory. Compared with existing methods, the method proposed by the application can effectively improve the reconstruction quality of images while maintaining the reconstruction speed.

Description

[Technical Field]

[0001] This invention discloses a compressed sensing image reconstruction method based on gated recursive units, belonging to the fields of signal processing and deep learning technology, specifically relating to a natural image reconstruction method applied to compressed sensing. [Background Technology]

[0002] Compressed sensing (CS) theory shows that when a signal is sparse in certain transform domains, it can be reconstructed with high probability from measurements with a dimension far less than that of the original signal. Combining the traditional signal sampling and compression process provides a new solution to the signal acquisition problem, greatly alleviating the need for high transmission bandwidth and large storage space, and realizing low-cost sensor data compression. Research on CS mainly focuses on two aspects: (1) designing an effective sampling matrix; (2) constructing a high-quality reconstruction solver to recover the high-dimensional original signal from low-dimensional measurements. The inversion process of CS is an NP-hard problem. When the image does not precisely satisfy sparsity in the transform domain, traditional CS image reconstruction methods usually suffer from poor reconstruction performance.

[0003] In recent years, due to the powerful learning capabilities of deep neural networks, they have developed rapidly in various fields and have been widely applied. A series of image CS methods based on deep networks have been proposed. These methods relax the assumptions about the sparsity of the original image and jointly optimize the sampling matrix and nonlinear recovery operator, enabling them to coordinate with each other through end-to-end training metrics. This allows them to learn the structural and textural features of the image more effectively, significantly improving the efficiency and quality of image CS reconstruction.

[0004] Generally, CS image reconstruction methods are divided into two categories: (1) Traditional CS reconstruction algorithms based on prior knowledge, including greedy algorithms, iterative algorithms, and convex relaxation algorithms, mainly utilize the sparsity of the signal and structural priors to find the unique true solution from an infinite number of possible solutions. However, this method is limited by some general priors, such as sparsity and some known support, so the reconstruction effect is generally poor. (2) Network-based CS reconstruction algorithms directly map the original signal and the reconstructed signal, and use the powerful learning ability of deep neural networks to reconstruct the signal. Compared with the former, it runs faster and has better reconstruction performance. However, these purely data-driven algorithms are trained into a black box, and their lack of interpretability seriously hinders their application. Therefore, some CS reconstruction algorithms that combine the advantages of both have been proposed, called interpretable deep compressed sensing (DCS) reconstruction algorithms, which are reconstruction methods that combine prior knowledge and data-driven approaches. These algorithms can be broadly categorized into two types: one treats the neural network as a black box executing certain functions within the algorithm, known as the black-box neural network method. However, its performance improvement is usually attributed to the stacking of modules, resulting in high model complexity and training costs. The other type unfolds the iterative algorithm into a neural network, known as the Deep Unfolding Network (DUN) method. DUN has become a popular CS method due to its good interpretability and strong mapping ability. DUN typically consists of a certain number of sequential sub-stages, such as ISTA-Net and AMP-Net. It is easy to see that each stage in the network is mainly directly influenced by the previous stage. As the number of stages increases, the connections between distant stages become almost nonexistent, easily leading to feature loss. Therefore, effectively increasing the inter-stage connections is a crucial consideration for DUN. Furthermore, the construction of these algorithmic sub-stages often employs simple stacking of convolutional layers, making it difficult to fully utilize the image reconstruction capabilities of the proposed model.

[0005] Therefore, to address the aforementioned problems, this invention proposes a compressed sensing image reconstruction method based on gated recurrent units, termed Gated Recurrent Deep Unfolding Network (GRDU-Net). This invention innovatively designs an Enhanced Proximal Mapping Module (EPMM), integrating an intra-level attention mechanism based on Channel Attention Residual Blocks (CARBs) and an inter-level memory mechanism based on Conv Gated Recurrent Units (ConvGRUs), thus solving the problems of insufficient DUN representation power and inherent information loss during long-distance feature transmission. Experimental results show that, compared with existing methods, the proposed GRDU-Net effectively improves the reconstruction performance of CS images. [Summary of the Invention]

[0006] The purpose of this invention is to combine prior-based and network-based reconstruction models, proposing a compressed sensing image reconstruction method based on gated recurrent units (GRUs). This invention combines the advantages of both methods, making the reconstruction network interpretable while solving the information loss problem commonly encountered in DUNs due to long-distance transmission, thus achieving high-performance reconstruction of CS images.

[0007] Traditional image CS methods typically reconstruct the original image by solving the following optimization problem. :

[0008]

[0009] in Represents the measurement matrix. This represents the measured value after sampling. It is a regularization term The weighted hyperparameters, Let represent the prior terms. In the inverse solution of the CS problem, this expression is approximately represented as the following iterative convergence problem:

[0010]

[0011]

[0012] Iterative update and Until it converges. , and Let represent the output, intermediate variable, and weighted step size of the k-th iteration, respectively. In practical terms, the former represents gradient descent, and the latter represents the proximal mapping, utilizing the proximal operator. To solve this problem, in this invention, a deep neural network is used to solve the near-end mapping, and an unfolded network is constructed to achieve compressed sensing reconstruction.

[0013] GRDU-Net is a fully end-to-end learnable network that jointly optimizes the sampling and reconstruction networks. A proximal mapping module is designed to achieve persistent storage and transmission of long short-term feature memory, ensuring image reconstruction quality and offering advantages such as fast, accurate image reconstruction and clear interpretability. Addressing the severe energy constraints common in wireless sensor networks, compressed sensing provides a novel solution. Compressed sensing combines sampling and compression processes, allowing for high-quality reconstruction of original data from only one-tenth or even one-hundredth of the data volume. If GRDU-Net is deployed in a wireless sensor network environment, it can reduce data acquisition redundancy, lower data transmission pressure and energy consumption, thereby extending network lifetime. This is a good solution to alleviate the major problems faced by wireless sensor networks.

[0014] The objective of this invention is achieved through the following technical solution:

[0015] (1) Dataset Acquisition: Randomly extract data of varying sizes from the Set91 natural image dataset. Image patches, the dataset is denoted as .

[0016] (2) Network parameters: Compressed sensing sampling rate Maximum number of training sessions epoch Number of samples selected in one training session batch_size Learning rate Step length and the number of reconstructed network sub-stages N s wait.

[0017] (3) Adaptive sampling: Adaptive sampling is achieved using convolution. Replaced with M A convolutional kernel of size 1, with 1 channel, based on That is, the measured value is obtained. .

[0018] (4) Initial Restructuring: Based on Image initialization and reconstruction are achieved using the transpose convolution of the sampling matrix. N Each convolutional kernel is 1×1 in size, with 1 channel. M After convolution, we obtain 1×1× NPerforming a PixelShuffle operation on the tensor yields... .

[0019] (5) Deep Reconstruction: Intra-level attention mechanism and inter-level memory mechanism were designed and integrated into the long short-term memory enhancement proximal mapping module, realizing the construction of a deep unfolded network. Input deep reconstruction network, through N s Each sub-stage output yields... .

[0020] (6) Calculate the loss function: ,in For the first i Zhang training images, For the first i Reconstructed images from training images, The total number of training images, This represents the total number of pixels in each image. If binary constraints are added to the measurement matrix during training, then the measurement matrix... Add orthogonalization constraints, ,Right now .

[0021] (7) Training: Determine whether the maximum has been reached. epoch If the value is not reached, error backpropagation is performed, and an adaptive learning rate optimization algorithm is used. Update the weights, and when the target is reached, save the network parameters and end the training.

[0022] (8) Evaluation: Input the benchmark dataset images into the trained deep reconstruction network to obtain the reconstructed images, and measure the reconstruction performance by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) between the original images and the reconstructed images.

[0023] [Advantages and positive effects of the present invention]

[0024] Compared with the prior art, the present invention has the following advantages and positive effects:

[0025] First, this invention combines prior-based and network-based reconstruction models to propose a fully end-to-end learning, interpretable, and persistent long short-term memory deep unfolded network, which achieves joint optimization of adaptive sampling and restoration of CS images.

[0026] Second, by utilizing the measurement matrix and its transpose to achieve adaptive image sampling and initialization reconstruction, the number of network training parameters and computational costs are significantly reduced. Furthermore, binary constraints can be added to the measurement matrix to facilitate storage and hardware implementation.

[0027] Third, an innovative Long Short-Term Feature Memory Enhanced Proximal Mapping Module (EPMM) was designed, constructing two mechanisms: intra-level attention and inter-level memory. By utilizing channel attention and long-term memory flow, the inherent information loss problems that occur during long-distance transmission of feature information in existing deep unfolded networks, such as insufficient representational power, are addressed, thus achieving persistent storage and transmission of long short-term feature memory. Experiments have demonstrated that this invention exhibits good stability and excellent reconstruction performance on the test set. [Attached Image Description]

[0028] Figure 1 This is a flowchart of the compressed sensing image reconstruction method based on gated recursive units proposed in this invention;

[0029] Figure 2 This is a structural diagram of the deep reconstruction network of this invention;

[0030] Figure 3 This is a diagram of the CARB structure;

[0031] Figure 4 This is a diagram of the ConvGRU architecture.

Detailed Implementation Methods

[0032] To make the embodiments and advantages of the present invention clearer, the present invention will be described in more detail below with reference to the accompanying drawings and examples.

[0033] (1) Input: Training dataset , The image patch size is 33×33.

[0034] (2) Initialization: Compressed sensing sampling rate Maximum number of training sessions epoch =150, Number of samples selected in one training session batch_size =64, Learning Rate Step length Initialize the iteration depth k=1, with an upper limit equal to the number of sub-stages in the reconstructed network. N s =9.

[0035] (3) Adaptive sampling: for Each convolutional kernel is 33×33 in size, with 1 channel. The measured value can then be obtained. .

[0036] (4) Initial Restructuring: Based on Image initialization and reconstruction are achieved using transpose convolution of the sampling matrix, employing 1089 1×1 convolution kernels with a kernel channel size of [missing information]. M After convolution, we obtain 1×1× N Performing a PixelShuffle operation on the tensor yields... .

[0037] (5) Deep Reconstruction: This part of the network consists of... N s The process consists of repeating stages, each including gradient descent and the proximal mapping module EPMM. Therefore, the k-th stage can be represented as... EPMM includes an intra-level attention mechanism based on Channel Attention Residual Blocks (CARBs) and an inter-level memory mechanism based on Conv Gated Recurrent Units (ConvGRUs). Input deep reconstruction network, through N s Each sub-stage output yields... .

[0038] (6) Calculate the loss function: ,in For the i-th training image, The reconstructed image of the i-th training image, The total number of training images, This represents the total number of pixels in each image. If binary constraints are added to the measurement matrix during training, then the measurement matrix... Add orthogonalization constraints, ,Right now .

[0039] (7) Training: Determine whether the maximum has been reached. epoch If the value is not reached, error backpropagation is performed, and an adaptive learning rate optimization algorithm is used. Update the weights, and when the target is reached, save the network parameters and end the training.

[0040] (8) Evaluation: Input the benchmark dataset images into the trained deep reconstruction network to obtain the reconstructed images, calculate PSNR and SSIM, and measure the reconstruction performance.

[0041] Specifically, the learnable parameter set in GRDU-Net Including measurement matrix Step length Transformation parameters involved in deep reconstruction networks , , , , , ,Right now Note that, except for all stages in the deep reconstruction network... They share the same parameters. , , This represents three convolution operations. , This indicates the CARB (Carbon Buffer) block representing the residuals of two channels. This represents the ConvGRU (Convolutional Gated Recursive Unit). It has one input channel and C output channels. It has C+1 input channels and C output channels. It has C input channels and one output channel, after , and The number of channels remains unchanged at C.

[0042] The simulation experiment of this invention was run under the simulation conditions of GPU: NVIDIA Quadro RTX 4000, Ubuntu 20.04 (64 bit) operating system, and development platform: Visual Studio Code, PyTorch 1.11.

[0043] In algorithm testing, two widely used benchmark datasets were used: Set11 and Urban100. For ease of comparison, grayscale images were used for performance comparison. Benchmark experiments included ISTA-Net+, DPA-Net, OPINE-Net, and AMP-Net. The average PSNR and SSIM of each algorithm's reconstructed dataset were compared. A higher PSNR value and an SSIM value closer to 1 indicate more accurate image reconstruction quality and a closer resemblance to the original image. In all unfolding algorithms, the number of stages was set to 9.

[0044] In this invention, the sampling rate is set. The parameter selection has a certain degree of universality; the same set of parameters was used for the test set in this experiment, and this parameter set can also be extended to the reconstruction process of other natural images. Table 1 shows the average time for different algorithms to reconstruct a 256×256 image, with a speed of approximately 80-81 FPS, which is much higher than the smooth playback frame rate of high-definition video, achieving real-time image reconstruction. This has a significant advantage for tasks with limited processing resources and high time costs. Tables 2 and 3 show the PSNR and SSIM of datasets Set11 and Urban100 at different measurement rates, respectively. The experimental results show that the method proposed in this invention achieves the best reconstruction effect at all sampling rates, especially at lower sampling rates. More specifically, for dataset Set11, with a sampling rate of 0.01, the proposed method GRDU-Net, compared to ISTA-Net+, DPA-Net, OPINE-Net, and AMP-Net algorithms, shows gains of 3.14 dB, 2.29 dB, 0.67 dB, and 1.63 dB in average PSNR, respectively, with a minimum increase of 3.4%. In terms of average SSIM, the methods improve by 0.1358, 0.0608, 0.0256, and 0.0731, respectively, with a minimum increase of 4.7%. The data in the table also reveals an interesting phenomenon: when using binary matrix constraints, better reconstruction performance can be obtained at low sampling rates. However, as the sampling rate increases, GRDU-Net, using floating-point matrices, exhibits better reconstruction performance. This phenomenon will be verified in future experiments. In summary, the proposed method demonstrates high image reconstruction quality and good stability, making it an effective compressed sensing image reconstruction method.

[0045] Table 1. Average time (s) for each method to reconstruct a 256×256 image

[0046]

[0047] Table 2. Comparison of PSNR (dB) and SSIM∈[0,1] for reconstructing Set11 by different methods

[0048]

[0049] Table 3. Comparison of PSNR (dB) and SSIM∈[0,1] for Urban100 reconstruction by different methods

[0050]

Claims

1. A compressed sensing image reconstruction method based on gated recursive units, comprising the following steps: Step 1: Dataset Acquisition: Randomly extract data of varying sizes from the Natural Image Dataset Set91. Image patches, the dataset is denoted as ; Step 2: Network parameters: Compressed sensing sampling rate Maximum number of training sessions Number of samples selected in one training session Learning rate Step length and the number of reconstructed network sub-stages ; Step 3: Adaptive Sampling: Adaptive sampling is achieved using convolution. Replaced with indivual A convolutional kernel of size 1, with 1 channel, based on That is, the measured value is obtained. ; Step 4: Initialize Refactoring: Based on Image initialization and reconstruction are achieved using the transpose convolution of the sampling matrix. Each convolutional kernel is 1×1 in size, with 1 channel. After convolution, obtain Performing a PixelShuffle operation on the tensor yields... ; Step 5: Deep Reconstruction: Intra-level attention and inter-level memory mechanisms were designed and integrated into the long short-term memory enhanced proximal mapping module, realizing the construction of a deep unfolded network. Specifically, this deep reconstruction network consists of N s The process consists of repeating stages, each including gradient descent and the proximal mapping module EPMM. Therefore, the k-th stage can be represented as... EPMM includes an intra-level attention mechanism based on channel residual attention blocks (CARB) and an inter-level memory mechanism based on convolutionally gated recurrent units (ConvGRU); Input deep reconstruction network, through Each sub-stage output yields... ; Step 6: Calculate the loss function: ,in For the first Zhang training images, For the first Reconstructed images from training images, The total number of training images, The total number of pixels in each image; If binary constraints are added to the measurement matrix during training, then the measurement matrix... Add orthogonalization constraints, ,Right now ; Step 7: Training: Determine if the maximum has been reached. If the value is not reached, error backpropagation is performed, and an adaptive learning rate optimization algorithm is used. Update the weights; if the target is reached, save the network parameters and end the training. Step 8: Evaluation: Input the benchmark dataset images into the trained deep reconstruction network to obtain reconstructed images, and measure the reconstruction performance by calculating the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) between the original and reconstructed images.

Images