An image compression sensing reconstruction method and system based on an optimization algorithm
By combining traditional algorithms and deep learning, and using the GSISTA-Net network for image compressed sensing, convolutional sampling, skip information connections, and dual-scale denoising are employed to address the performance limitations of deep learning in the absence of training data and the high computational complexity of traditional algorithms, thereby achieving high-quality image reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUBEI UNIV OF TECH
- Filing Date
- 2023-03-08
- Publication Date
- 2026-06-26
AI Technical Summary
Existing deep learning compressed sensing technology performs poorly in the absence of a large amount of high-quality training data. Furthermore, traditional compressed sensing algorithms have high computational complexity and limited reconstruction accuracy, and suffer from information loss and incomplete feature fusion during information flow transmission.
Combining traditional algorithms and deep learning, a GSISTA-Net network based on an iterative shrinking threshold generalization structure is proposed. It adopts a convolutional sampling and skip information connection structure, designs a dual-scale denoising module, and learns parameters through backpropagation to achieve image sampling and reconstruction.
It significantly improves image reconstruction quality, has interpretability, and achieves a good balance between computational complexity and reconstruction accuracy, outperforming existing methods.
Smart Images
Figure CN117495988B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and mainly relates to a method and system for image compressed sensing reconstruction based on optimization algorithms. Background Technology
[0002] Compressed sensing theory suggests that when a signal is sparse, it can be compressed and sampled with fewer samples than the traditional Nyquist sampling theorem, and then the original signal can be reconstructed by solving a nonlinear optimization problem. Traditional compressed sensing reconstruction algorithms have a theoretical basis, but their computational complexity is high and their reconstruction accuracy is limited. In recent years, the excellent performance of deep learning in advanced image processing tasks has attracted the attention of researchers. Deep learning based compressed sensing (DLCS) has opened a new door for image compressed sensing technology, effectively overcoming the shortcomings of traditional compressed sensing algorithms to some extent. DLCS network models are generally composed of multiple network layers and activation functions stacked together; the most common types include denoising autoencoders, convolutional neural networks, and multi-scale residual reconstruction networks. This method can not only reconstruct images in real time, but its image reconstruction effect is also superior to traditional compressed sensing methods. However, the inherent black-box nature of deep learning technology makes DLCS lack a theoretical basis, and under purely data-driven conditions, the performance of deep learning technology depends on the quantity and quality of training data. Without a large amount of high-quality training data, the performance of deep neural networks will be lower than expected.
[0003] However, compared to previous DLCS methods, algorithmic unfolding networks are an interpretable DLCS approach. Algorithmic unfolding was first proposed in 2010 by Gregor and LeCun for solving sparse coding problems. They established the LISTA (LearnedISTA) model based on the Iterative Shrinkage and Threshold Algorithm (ISTA). Since then, algorithmic unfolding techniques have been widely used to solve many important problems in signal and image processing. Like traditional algorithms, algorithmic unfolding networks maintain the information flow in the pixel space between stages. However, during information flow transmission, only image features processed by the network are transmitted. The structure for transmitting information in pixel space does not fully utilize the information contained in the image features during forward propagation, thus leading to information loss and incomplete feature fusion. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides a method and system for image compressed sensing reconstruction based on optimization algorithms.
[0005] This invention combines the advantages of traditional algorithms and deep learning-based compressed sensing reconstruction methods, proposing an algorithm based on the Generalized Structure of Iterative Shrinkage Threshold (GSISTA) algorithm and the interpretable deep network GSISTA-Net derived from it for image CS sampling and reconstruction. In the sampling stage, this invention utilizes convolutional sampling instead of traditional random matrix sampling. In the reconstruction stage, a skip information connection structure is designed in the reconstruction submodule R, using a residual module to connect the preceding and following feature information, avoiding the inherent information loss in deep unfolded networks. Furthermore, a dual-scale denoising module is designed at the back end of the reconstruction submodule R, combining features at different scales to denoise the image. All parameters involved in GSISTA-Net (such as nonlinear transformation, shrinkage threshold, stride, etc.) are learned end-to-end through backpropagation.
[0006] The above-mentioned technical problems of the present invention are mainly solved by the following technical solutions:
[0007] A compressed sensing reconstruction method for images based on optimization algorithms, characterized by comprising:
[0008] Step 1: Input an image block of size B×B, use a single unbiased convolutional layer to simulate the matrix sampling y=Φx to obtain the measured value y, and then perform initial reconstruction;
[0009] Step 2: During the initial reconstruction, a single convolutional layer is used to convert Φ... T The pixels are reshaped into N filters, each with a kernel size of 1×1×M. After obtaining the features, the pixels are shuffled.
[0010] Step 3: Pixel shuffling reshapes the N×1×1 tensor into... Obtain the initial reconstructed image x 0 ;
[0011] Step 4: Initial reconstruction image x 0 As input, GSISTA is unfolded into a deep network to perform deep reconstruction based on convolutional operations, while the skip information connection structure is used to fuse the information of its input features with the information of its output features.
[0012] Step 5: Denoise the features fused in Step 4 at different scales, then fuse the denoised features to obtain the output features, and proceed to the next iteration;
[0013] Step 6: Repeat steps 4 and 5 until the final reconstructed image is obtained.
[0014] In the aforementioned image compressed sensing reconstruction method based on an optimization algorithm, specifically a deep network based on GSISTA, the method involves applying iterative shrinkage operators to the GSISTA algorithm. The threshold is updated using a linear combination of the first n predicted values. The GSISTA algorithm is expressed as follows:
[0015]
[0016]
[0017]
[0018]
[0019] Where k represents the number of iterations, ρ represents the step size, and λ represents the threshold parameter. When n = 0... The iterative algorithm is represented by ISTA. When n=1, the iterative algorithm is represented by FISTA. The generalized algorithm introduces the prediction results of the first n iterations as intermediate features into the image reconstruction network.
[0020] In the aforementioned image compressed sensing reconstruction method based on optimization algorithms, the deep reconstruction network consists of n identical R modules iteratively. Within each R module, a nonlinear transformation function F(·) is used to simulate the sparse transformation matrix Ψ. This nonlinear transformation function comprises two unbiased linear convolution operators.
[0021] F(x) = VReLU(Wx)
[0022] Where V and W represent two convolution operations, ReLU is a rectified linear unit, and inverse transform... Similar to the F(·) structure, but with different kernel parameter settings, it becomes after network unfolding.
[0023]
[0024] Where λ represents the learnable parameters in the iterative module, and in each reconstruction stage of the network, its transformation parameters and threshold parameters are... F k (·), λ k They are all different.
[0025] In the above-mentioned image compressed sensing reconstruction method based on optimization algorithm, a residual module is used in the skip information connection structure to pass the input features to the output features, and the information of the input features is processed and passed to each layer inside the residual module.
[0026] In the above-mentioned image compressed sensing reconstruction method based on optimization algorithms, the specific operation of dual-scale denoising is as follows:
[0027] The first scale denoises the output features in step 4 by passing them through different scales. The low-resolution features are downsampled by convolution with a stride of 2 to the input features. A residual module is used to denoise the low-resolution features, and then an upsampling operation is performed to restore the features to their original size.
[0028] The second scale uses a combination of convolution operators and ReLU to remove other noise, fuses the denoised features to obtain output features, and sends them to the next stage.
[0029] Finally, information flow is gradually realized in the feature space.
[0030] In the image compressed sensing reconstruction method based on the above optimization algorithm, the measured value y is used to perform initial reconstruction to obtain the initial reconstructed image x. 0 x 0 After performing steps 4 and 5 following the depth reconstruction, perform the next depth reconstruction, repeating steps 4 and 5 to obtain the final reconstructed image.
[0031] An image compressed sensing reconstruction system based on an optimization algorithm, characterized in that it includes:
[0032] Sampling module: This module takes an input image patch of size B×B and uses a single unbiased convolutional layer to simulate the matrix sampling process y = Φx to obtain a measured value y, which is then input into the initial reconstruction module.
[0033] Initial reconstruction module: uses a single convolutional layer to reconstruct Φ T The pixels are reshaped into N filters, each with a kernel size of 1×1×M, and the resulting features are input into the pixel shuffling operation module.
[0034] Pixel shuffling module: Reshapes a tensor of size N×1×1 into... Obtain the initial reconstructed image x 0 ;
[0035] Several depth reconstruction modules: initial reconstructed image x 0 As input to the deep reconstruction module, GSISTA is unfolded into a deep network, and convolutional operations are performed in the deep reconstruction submodule R. At the same time, the information of its input features and output features are fused using a skip information connection structure. The fused features are then denoised at different scales, and the denoised features are fused to obtain the output features, which are then fed into the next deep reconstruction module until all deep reconstruction modules have completed the reconstruction process and the final reconstructed image is output.
[0036] In the aforementioned system, the deep network based on GSISTA unfolds as follows: the GSISTA algorithm applies iterative operators to the iterative shrinkage operator. The threshold is updated using a linear combination of the first n predicted values. The GSISTA algorithm is expressed as follows:
[0037]
[0038]
[0039]
[0040]
[0041] Where k represents the number of iterations, ρ represents the step size, and λ represents the threshold parameter. When n = 0... The iterative algorithm is represented by ISTA. When n=1, the iterative algorithm is represented by FISTA. The generalized algorithm introduces the prediction results of the first n iterations as intermediate features into the image reconstruction network.
[0042] In the system described above, the deep reconstruction network consists of n identical R modules that iterate together. Within each R module, a nonlinear transformation function F(·) is used to simulate the sparse transformation matrix Ψ. This nonlinear transformation function is composed of two unbiased linear convolution operators.
[0043] F(x) = VReLU(Wx)
[0044] Where V and W represent two convolution operations, ReLU is a rectified linear unit, and inverse transform... Similar to the F(·) structure, but with different kernel parameter settings, it becomes after network unfolding.
[0045]
[0046] Where λ represents the learnable parameters in the iterative module, and in each reconstruction stage of the network, its transformation parameters and threshold parameters are... F k (·), λ k They are all different;
[0047] In the skip information connection structure, a residual module is used to pass the input features to the output features, and the information of the input features is processed and passed to each layer inside the residual module.
[0048] In the above system, the specific operation of dual-scale denoising is as follows:
[0049] The first scale denoises the output features in step 4 by passing them through different scales. The low-resolution features are downsampled by convolution with a stride of 2 to the input features. A residual module is used to denoise the low-resolution features, and then an upsampling operation is performed to restore the features to their original size.
[0050] The second scale uses a combination of convolution operators and ReLU to remove other noise, fuses the denoised features to obtain output features, and sends them to the next stage.
[0051] Finally, information flow is gradually realized in the feature space.
[0052] Therefore, the present invention has the following advantages:
[0053] 1. In each depth reconstruction submodule R, the skip connection structure and the dual-scale denoising module significantly improve the image reconstruction quality.
[0054] 2. The generalized form of the iterative shrinking threshold algorithm, GSISTA, introduces the results of the first n predictions as intermediate features into the algorithm and extends it into an image reconstruction network, giving the image reconstruction network a certain degree of interpretability. Attached Figure Description
[0055] Appendix Figure 1 This is a framework diagram of the image compressed sensing reconstruction method based on optimization algorithms of the present invention. Detailed Implementation
[0056] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.
[0057] Example:
[0058] ISTA is a gradient-based method where, in each iteration, the gradient of the differentiable term is projected and then thresholded to a specific value. The signal to be reconstructed is updated through this thresholding operation, and the specific iterative formula is as follows:
[0059] z k =x k-1 -ρΦ T (Φx k-1 -y)
[0060]
[0061] Here, k represents the number of iterations, ρ represents the step size, and λ represents the threshold parameter. However, ISTA converges slowly, and the optimal solution obtained is often not sparse enough. Therefore, many research works have attempted to accelerate the convergence speed of the algorithm to obtain a faster solution algorithm.
[0062] Unlike ISTA, which only uses the value from the previous iteration, FISTA obtains new iteration values by thresholding the values from the previous two iterations. The FISTA iteration steps are as follows:
[0063]
[0064]
[0065]
[0066]
[0067] Inspired by FISTA, a linear combination of the first n predicted values is used, and its corresponding formula is generalized to the following formula, thus obtaining the generalized structure of the iterative shrinking threshold algorithm (GSISTA):
[0068]
[0069] In the appendix Figure 1 As shown, GSISTA-Net takes image patches of the same size as input and outputs a reconstructed image through an initial reconstruction module and a depth reconstruction module. The depth reconstruction module consists of n depth reconstruction sub-modules R, each of which includes a GSISTA-based unfolded network, a skip connection structure, and a dual-scale denoising module. Their functions are described below.
[0070] 1) GSISTA-based unfolded network
[0071] The GSISTA algorithm applies iterative operators to the iterative shrinkage operator. Unlike traditional iterative threshold shrinkage methods, the GSISTA algorithm updates the threshold using a linear combination of the first n predicted values. The formula for the GSISTA algorithm is as follows:
[0072]
[0073]
[0074]
[0075]
[0076] Where k represents the number of iterations, ρ represents the step size, and λ represents the threshold parameter. When n = 0, The iterative algorithm is represented by ISTA. When n=1, the iterative algorithm is represented by FISTA. The generalized algorithm incorporates the predictions from the first n iterations as intermediate features into the image reconstruction network, making full use of feature information and improving the image reconstruction effect.
[0077] The deep reconstruction network consists of n identical R modules that iterate through each other. In each R module, a nonlinear transformation function F(·) is used to simulate the sparse transformation matrix Ψ in GSISTA. This function consists of two linear convolution operators with unbiased terms.
[0078] F(x) = VReLU(Wx)
[0079] Where V and W represent two convolution operations, and ReLU is a rectified linear unit. Inverse transform Similar to the F(·) structure, but with different convolution kernel parameters. Therefore, after unfolding the network, it becomes...
[0080]
[0081] Where λ represents the learnable parameters in the iterative module, and in each reconstruction stage of the network, its transformation parameters and threshold parameters are... F k (·), λ k They are all different.
[0082] 2) Jump connection structure
[0083] The skip connection structure utilizes a residual module to pass input features to output features, and processes the input feature information before passing it to each layer within the residual module. This skip connection structure fully leverages the hierarchical information of each layer, achieving a network information sharing mechanism during forward propagation between layers. Later convolutional layers in the network structure not only acquire information from forward propagation but also obtain more information from lower levels.
[0084] 3) Dual-scale noise reduction module
[0085] The dual-scale denoising module downsamples the input features using a 2-stride convolution, then uses a residual module to denoise the low-resolution features, followed by an upsampling operation to restore the features to their original size. The other scale uses a combination of convolution operators and ReLU to remove other noise. The denoised features are then fused to obtain the output features, which are fed into the next stage. Finally, information flow is gradually realized in the feature space.
[0086] Through the combined action of a GSISTA-based unfolded network, a skip connection structure, and a dual-scale denoising module, the hierarchical information of each layer is fully utilized, and the image is effectively denoised. The reconstruction network is based on algorithmic unfolding, giving the network itself a theoretical basis. Furthermore, the later convolutional layers in the network structure not only acquire information from the forward propagation but also acquire more low-level information.
[0087] The measured value y is input into the depth reconstruction module. Steps 4 and 5 are performed in the depth reconstruction module and output to the next depth reconstruction module. The above operations are repeated to obtain the final reconstructed image, which significantly improves the image reconstruction quality.
[0088] The positive effects of this invention will be further described below with reference to specific experimental data.
[0089] 1) Analyze the impact of the number of iterations in the R module on the image reconstruction results.
[0090] Table 1 shows the impact of the number of iterations in module R on image reconstruction results.
[0091]
[0092] In this section, the invention will explore the impact of the number of R module iterations (including 5, 7, 9, and 11 iterations) on the results. Table 1 shows that the number of module iterations has a certain impact on network performance. By balancing performance and computational complexity, nine stages were used in this experiment, which also demonstrates the effectiveness of the iterative network design.
[0093] 2) Analyze the impact of different modules on image reconstruction results.
[0094] To avoid information loss and incomplete information fusion during deep reconstruction, a Jump Information Connection (JIC) structure is introduced. Furthermore, for better denoising performance, a dual-scale denoising (DSD) module is designed within the deep reconstruction submodule R. To verify the effectiveness of the network modules, this invention uses the concept of controlled variables to divide the network into a basic network without JIC and DSD, a network with only JIC, a network with only DSD, and a network with both JIC and DSD modules—that is, the method proposed in this paper.
[0095] Table 2. Impact of different modules on image reconstruction results
[0096]
[0097] As shown in Table 2, the networks of each module in this invention are represented as "Base-Net, Base-Net-JIC, Base-Net-DSD, and GSISTA-Net". It can be observed that the PSNR / SSIM of Base-Net-JIC is improved by 0.14 / 0.0031 compared to Base-Net, indicating that the JIC module contributes to the image reconstruction effect. The PSNR / SSIM of Base-Net-DSD is improved by 0.11 / 0.0020 compared to Base-Net, indicating that the DSD module improves the network's denoising effect, thus resulting in better image reconstruction.
[0098] 3) Investigate the influence of the value of n in the generalization formula on image reconstruction.
[0099] Table 3 shows the influence of the value of n on image reconstruction in the generalization formula.
[0100]
[0101] In the generalization formula above, different values of n result in different iterative algorithm expressions. As shown in Table 3, when n is 0, the iterative algorithm is represented as ISTA, and the PSNR / SSIM of the network reconstruction result is 29.65 / 0.8883. When n is 1, the iterative algorithm is represented as FISTA, and the PSNR / SSIM of the network reconstruction result is 29.78 / 0.8905. When n is the number of network iterations, the PSNR / SSIM of the network reconstruction result is 29.91 / 0.9034. This indicates that the choice of the value of n in the iterative algorithm has a certain impact on the reconstruction effect of the algorithm.
[0102] 4) Verify the effectiveness of the GSISTA-Net network framework.
[0103] To verify the effectiveness of the proposed algorithm's expanded network GSISTA-Net framework, this invention was compared with five state-of-the-art image compressed sensing methods, whose sampling matrices are also learnable, including AdapRecon, CSNet, and ISTA-Net. + AMPNet and OPINE-Net + Extensive experiments have demonstrated the advantages of the proposed image reconstruction method in terms of image quality and visualization results, as shown in Table 4.
[0104] Table 4 Comparison of average PSNR / SSIM performance of different CS methods on dataset Set11
[0105]
[0106] As shown in Table 4, GSISTA-Net outperforms all other methods across all CS sampling rates. It is noteworthy that the three algorithmic unfolding methods mentioned above—ISTA-Net+, AMPNet, and OPINE-Net+—significantly outperform the ordinary neural networks AdapRecon and CSNet in terms of reconstruction quality. This also indicates that algorithmic network unfolding improves the performance of image reconstruction networks to some extent.
[0107] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. An image compressed sensing reconstruction method based on optimization algorithms, characterized in that, include: Step 1: Input an image patch of size B×B, and simulate matrix sampling using a single unbiased convolutional layer. Obtain the measured value y Then, initial reconstruction was carried out; Step 2: During the initial reconstruction, a single convolutional layer is used to... Remodeling N There are 1 filter, each filter kernel size is 1. After obtaining the features, the pixels are shuffled. Step 3, Pixel Shuffling: The size is... The tensor is reshaped into Obtain the initial reconstructed image ; Step 4: Initial Reconstruction Image As input, GSISTA is unfolded into a deep network to perform deep reconstruction based on convolutional operations, while the skip information connection structure is used to fuse the information of its input features with the information of its output features. Step 5: Denoise the features fused in Step 4 at different scales, then fuse the denoised features to obtain the output features, and proceed to the next iteration; Step 6: Repeat steps 4 and 5 until the final reconstructed image is obtained; Deep networks based on GSISTA expansion, specifically, involve applying iterative operators to the GSISTA algorithm, specifically the iterative shrinkage operator. Above, adopt the previous n The threshold is updated using a linear combination of predicted values. The GSISTA algorithm is expressed as follows: in Represents the number of iterations. Represents step length, This represents the threshold parameter; when n=0, The iterative algorithm is represented by ISTA. When n=1, the iterative algorithm is represented by FISTA. The generalized algorithm will... n The predicted values from the previous step are used as intermediate features and introduced into the image reconstruction network. Deep reconstruction network by n A structure with the same R Module iterative composition, in R In the module, a nonlinear transformation function is used. To simulate sparse transformation matrices The nonlinear transformation function consists of two linear convolution operators with unbiased terms. in , These represent two convolution operations. For rectified linear units, inverse transform and With similar structures but different convolutional kernel parameter settings, the network becomes after being unfolded. in These are the learnable parameters in the iterative module, and their transformation parameters and threshold parameters at each reconstruction stage of the network. , , They are all different.
2. The image compressed sensing reconstruction method based on optimization algorithm as described in claim 1, characterized in that, In the skip information connection structure, a residual module is used to pass the input features to the output features, and the information of the input features is processed and passed to each layer inside the residual module.
3. The image compressed sensing reconstruction method based on optimization algorithm as described in claim 1, characterized in that, The specific operation of dual-scale denoising is as follows: The first scale denoises the output features in step 4 by passing them through different scales. The low-resolution features are downsampled by convolution with a stride of 2 to the input features. A residual module is used to denoise the low-resolution features, and then an upsampling operation is performed to restore the features to their original size. The second scale uses convolution operators and... ReLU The combined structure removes other noise, the denoised features are fused to obtain the output features, and then sent to the next stage; Finally, information flow is gradually realized in the feature space.
4. The image compressed sensing reconstruction method based on optimization algorithm as described in claim 1, characterized in that, Measured values y Initial reconstruction is performed to obtain the initial reconstructed image. , After performing steps 4 and 5 following the depth reconstruction, perform the next depth reconstruction, repeating steps 4 and 5 to obtain the final reconstructed image.
5. An image compressed sensing reconstruction system based on an optimization algorithm, characterized in that, include: Sampling module: Used to simulate matrix sampling using a single unbiased convolutional layer on an input image patch of size B×B. The measured value is obtained after the process. y Input into the initial reconstruction module; Initial reconstruction module: uses a single convolutional layer to... Remodeling N There are 1 filter, each filter kernel size is 1. The obtained features are input into the pixel shuffling operation module; Pixel shuffling module: Shuffles cards of size [size missing]. The tensor is reshaped into Obtain the initial reconstructed image ; Several depth reconstruction modules: initial reconstructed image As input to the deep reconstruction module, GSISTA is unfolded into a deep network, and then used in the deep reconstruction submodule. R Convolutional operations are performed in the process, and the information of the input features is fused with the information of the output features using the skip information connection structure. The fused features are then denoised at different scales, and the denoised features are fused to obtain the output features, which are then fed into the next depth reconstruction until all depth reconstruction modules have completed the reconstruction process and the final reconstructed image is output. Among them, the deep network based on GSISTA unfolds specifically as follows: the GSISTA algorithm applies iterative operators to the iterative shrinkage operator. Above, adopt the previous n The threshold is updated using a linear combination of predicted values. The GSISTA algorithm is expressed as follows: in Represents the number of iterations. Represents step length, This represents the threshold parameter; when n=0, The iterative algorithm is represented by ISTA. When n=1, the iterative algorithm is represented by FISTA. The generalized algorithm will... n The predicted values from the previous step are used as intermediate features and introduced into the image reconstruction network. Deep reconstruction network by n A structure with the same R Module iterative composition, in R In the module, a nonlinear transformation function is used. To simulate sparse transformation matrices The nonlinear transformation function consists of two linear convolution operators with unbiased terms. in , These represent two convolution operations. For rectified linear units, inverse transform and With similar structures but different convolutional kernel parameter settings, the network becomes after being unfolded. in These are the learnable parameters in the iterative module, and their transformation parameters and threshold parameters at each reconstruction stage of the network. , , They are all different.
6. The system according to claim 5, characterized in that, The specific operation of dual-scale denoising is as follows: The first scale denoises the output features in the depth reconstruction module through different scales. The low-resolution features are downsampled by the input features through a convolution with a stride of 2. A residual module is used to denoise the low-resolution features, and then an upsampling operation is performed to restore the features to their original size. The second scale uses convolution operators and... ReLU The combined structure removes other noise, the denoised features are fused to obtain the output features, and then sent to the next stage; Finally, information flow is gradually realized in the feature space.