Interpretability Compressed Sensing Image Reconstruction Method Based on Boundary Constraints

Abstract

The method for reconstructing an interpretable compressive sensing image based on boundary constraints of the present invention belongs to the field of image quality enhancement, solves the problem of compressive sensing image restoration. First, a compressive sensing reconstruction model with boundary constraints is designed, and the compressive sensing reconstruction problem based on boundary constraints is transformed into two sub-problems, and the two sub-problems are solved by means of iterative optimization; secondly, the iterative optimization process is expanded into an edge-guided interpretable image compressive sensing reconstruction network, and the edge-aware feature extraction module, the edge-guided intermediate variable update module, and the intermediate variable-guided image reconstruction module of the network are used to optimize the edge features, intermediate variable features, and reconstruction features of the image respectively; finally, a multi-memory enhancement mechanism is used to explore the feature dependencies at different iterative stages of the EGINet, and at the same time, a parallel cross-fusion module is used to effectively select the edge features and image features. The reconstruction performance of the present invention is superior to other image compressive sensing reconstruction methods.

Description

Technical Field

[0001] The present invention belongs to the field of image quality enhancement, and particularly relates to an interpretable compressive sensing image reconstruction method based on boundary constraints. Background Art

[0002] With the advent of the digital media era, a large number of information represented by digital signals has emerged. As digital information increases exponentially, the acquisition, processing, and storage of information face great challenges. The Nyquist sampling theorem is one of the most classic signal processing methods and has been regarded as an established rule in the signal processing field since it was proposed. However, this theorem has certain limitations. It cannot accurately recover the original signal at too low a sampling rate, and at too high a sampling rate, the amount of sampled data will be too large. Thus, traditional signal processing technologies can no longer meet the needs of humans, and it is urgent to find a more effective digital signal processing technology. To solve this problem, Donoho et al. proposed the Compressed Sensing (CS) theory, which breaks through the bottleneck of the Nyquist sampling theorem. This theory compresses data appropriately while acquiring signals, and its sampling frequency is much lower than the Nyquist sampling frequency. Its advantages are reducing the sampled data, saving storage space, and collecting sufficient information at the same time. Since this theory was proposed, it has attracted the attention of various fields in academia, including image denoising, image reconstruction, and data compression, and has been widely studied and applied in fields such as geological exploration, radar monitoring, medical detection, pattern recognition, single-pixel imaging, and radio communication.

[0003] The CS theory shows that: if a signal is sparse (i.e., compressible) in a certain orthogonal space, the signal can be sampled at a lower frequency (much lower than the Nyquist sampling frequency) and may be accurately reconstructed with high probability. Mathematically speaking, given the original natural signal x and the linear random projection matrix Φ, where Φ ∈ R M×N , the purpose of CS reconstruction is to infer the original signal x ∈ R M from the random CS observations y = Φx ∈ R N . When M << N, this inverse problem is an ill-posed and ill-conditioned problem. Traditional CS methods usually use sparsification operators of some predefined transform bases to represent the prior term, such as the Discrete Cosine Transform (DCT) and the Wavelet Transform (DWT). In most cases, they have the advantages of strong convergence and theoretical analysis, but their computational complexity is very high, and they often face difficulties in selecting the best transform and parameters.

[0004] Existing CS reconstruction methods are generally divided into the following categories: 1) Traditional compressive sensing reconstruction methods based on optimization; 2) Compressive sensing reconstruction methods based on deep networks; 3) Compressive sensing reconstruction methods based on interpretable models.

[0005] Traditional compressive sensing reconstruction methods based on optimization: Given a linear observation y = Φx (where x is the original signal and Φ is a linear random projection matrix), traditional compressive sensing reconstruction methods based on optimization reconstruct the original signal x by solving the following optimization problem:

[0006]

[0007] where F(·) is a non-linear transformation function used to sparsely represent natural images, and λ is a regularization parameter that controls the sparse term. Scholars usually use optimization-based methods to solve the optimization problem shown in the above formula. Its first term is the data fidelity term, and the second term is the regularization term. Commonly used optimization algorithms include the proximal gradient descent (PGD) algorithm, the half-quadratic splitting (HQS) algorithm, and the alternating direction method of multipliers (ADMM). These methods usually decouple the data fidelity term and the regularization term, and then alternately iterate to solve the data sub-problem and the prior sub-problem. However, all these optimization-based CS reconstruction methods usually require hundreds of iteration steps, with high computational complexity, and the selected image priors and regularization parameters are usually set manually. These fatal drawbacks limit the practical application of CS.

[0008] Compressive sensing reconstruction methods based on deep networks: Inspired by the powerful learning ability of deep neural networks, many compressive sensing reconstruction methods based on deep networks have been proposed recently. For example, Mousavi et al. proposed using a stacked denoising autoencoder (SDA) as an unsupervised feature learner to learn the structured representation from the training data and reconstruct the test data from the CS observation data. SDA can capture the statistical dependencies between different elements of the signal, thereby improving the signal recovery performance. However, the fully connected network (FCN) used by SDA requires a large number of learnable parameters. To alleviate this problem, scholars have proposed many reconstruction methods based on convolutional neural networks. Kulkarni et al. proposed a cascaded convolutional neural network architecture called ReconNet, which takes the CS observations of image patches as input and outputs the reconstructed image patches, improving the quality of the reconstructed image with less computational cost. However, the block-by-block reconstruction used by this algorithm often causes serious block artifacts in the image. Yao et al. proposed a residual reconstruction network (DR 2-Net) to reconstruct CS observation images. However, since this method still uses the direct mapping from block observations to corresponding image blocks, block artifacts still exist in the reconstructed images. To further remove the block artifacts in CS reconstruction, Shi et al. proposed a scalable convolutional neural network called SCSNet, which can achieve scalable sampling and reconstruction with only one model. SCSNet uses a block-by-block sampling and joint reconstruction method to effectively remove block artifacts. Compared with traditional CS reconstruction methods based on optimization, the main feature of CS reconstruction methods based on deep networks is that their networks are non-iterative, greatly reducing the time complexity of the algorithm. However, the networks used in these methods are usually trained as a black box, making the networks of these methods lack interpretability.

[0009] Compressed Sensing Reconstruction Methods Based on Interpretive Models: Compressed sensing reconstruction methods based on interpretive models combine the advantages of optimization-based methods and deep network-based methods. Deep neural networks and iterative optimizers can usually be integrated with each other, enabling the networks designed by them to be interpretable. For example, inspired by the Iterative Shrinkage Thresholding Algorithm (ISTA), Zhang et al. expanded this algorithm into the ISTA-Net network. Its advantage lies in that: compared with the manual setting of parameters in the ISTA algorithm, all parameters of the ISTA-Net reconstruction network are learnable. However, since the sampling method and reconstruction of the ISTA-Net network are both block-by-block, the performance of this network is limited. To improve CS reconstruction performance, Zhang et al. also proposed an enhanced version of OPINE-Net +, this network uses a learnable sampling matrix and incorporates both orthogonal constraints and binary constraints into the sampling matrix. At the same time, an inter-block correlation strategy is added to effectively remove block artifacts, thereby improving the quality of image compressive sensing reconstruction. During the feature transfer process between stages, there is a problem of information loss in existing compressive sensing reconstruction networks. To this end, Song et al. proposed a network with persistent memory, called the Memory-Augmented Deep Unfolding Network (MADUN), which incorporates two memory mechanisms, namely high-throughput short-term memory (HSM) and cross-stage long-term memory (CLM), effectively reducing information loss between different stages and enhancing the representation ability of the network. Different from the MADUN network, Zhang et al. pointed out that the input and output of each stage of the existing interpretable network are single-channel images, which limits the transmission of image information. Therefore, they proposed a high-throughput deep unfolding network, called HiTDUN, which can transmit multi-channel information between stages. In addition, Zhang et al. proposed a deep unfolding model called AMP-Net by unfolding the iterative denoising process of the approximate message passing algorithm. Considering that the sampling matrix of existing CS methods can only be used for specific models and has poor generalization ability, You et al. proposed a method for arbitrary sampling in compressive sensing (COAST), which introduces a random projection enhancement strategy (RPA) to achieve arbitrary sampling and uses one model to solve the CS problem of arbitrary sampling matrices. Considering that traditional algorithms only update images in the pixel space and do not make full use of the feature information of images, Chen et al. proposed a network with optimized feature space, called FSOINet, which maps the algorithm to the feature space to effectively utilize the convolutional features of images. Mou et al. integrated the gradient estimation strategy into the recent gradient descent algorithm and unfolded it into DGUNet to solve the image restoration problem. Different from the above methods that only use local convolutional features, Cui et al. incorporated non-local self-similar prior information into the compressive sensing reconstruction network to improve the reconstruction quality of images, proposed a non-local neural network, and called it NL-CSNet. This network adds non-local information in the observation domain and multi-scale feature domain to enhance the expression ability of the network. Although these methods have better interpretability by unfolding the optimization-based iterative solver onto the network, these algorithms ignore the importance of image edge information, and ultimately these methods often fail to achieve good performance at the boundaries of compressive sensing reconstructed images. Summary of the Invention

[0010] The present invention proposes an interpretable compressive sensing image reconstruction method based on boundary constraints, which consists of three parts: a sampling sub-network, an initial reconstruction sub-network, and a deep reconstruction sub-network. Here, the deep reconstruction sub-network includes three modules: an edge-aware feature extraction module (EAFM), an edge-guided intermediate variable update module (EGIM), and an intermediate variable-guided image reconstruction module (IGRM). In addition, a parallel cross-fusion module (PFM) is defined to fuse and reconstruct the image and the edge map using these modules, so that the reconstructed image has clearer edges and details. In addition, the network can well solve the problem that the existing compressive sensing reconstruction model is not interpretable and the problem that the reconstructed image has blurred boundaries.

[0011] To achieve the above object, the present invention is implemented by the following technical solutions.

[0012] The interpretable compressive sensing image reconstruction method based on boundary constraints includes the following steps:

[0013] Step 1) Training data construction

[0014] Before network training, the T91 dataset is used as the training dataset for the network;

[0015] Step 2) Constructing a compressive sensing reconstruction model based on boundary constraints

[0016] First, the image x is convolved with an edge convolution operator to obtain an edge map f(x); then, the Bayesian formula is used to represent the relationship between the image x, the edge map f(x), and the observation value y; finally, it is converted into a compressive sensing reconstruction model based on boundary constraints;

[0017] Step 3) Optimizing the compressive sensing reconstruction model with boundary constraints

[0018] The half-quadratic splitting (HQS) algorithm and the proximal gradient descent (PGD) algorithm are used to optimize the compressive sensing reconstruction model with boundary constraints obtained in step 2), and it is unfolded into a network;

[0019] Step 4) Sampling sub-network

[0020] The input image x is sampled using a sampling matrix to obtain the observation value y of the image;

[0021] Step 5) Initial reconstruction sub-network

[0022] The observation value y obtained in step 4) is deconvolved to achieve a preliminary reconstruction of the image x, obtaining an initial reconstructed image x (0) , and then from the initial reconstructed image x (0) the intermediate variable H is obtained (0) and the initial edge map f (0);

[0023] Step 6) Depth Reconstruction Sub-network

[0024] Expand the optimized model obtained in Step 3) into a depth reconstruction sub-network, which includes three modules: Edge-Aware Feature Extraction Module (EAFM), Edge-Guided Intermediate Variable Update Module (EGIM), and Intermediate Variable-Guided Image Reconstruction Module (IGRM). They are used to optimize the edge features, intermediate variable features, and reconstruction features of the image, respectively. [[ID=⑧]]

[0025] Step 7) Loss Function

[0026] The loss function consists of the reconstruction error loss of the image and the reconstruction error loss of the edge map, and the L1 norm is used to constrain the reconstruction error loss.

[0027] Furthermore, in Step 1), the T91 dataset containing 91 images is first used, and 88,912 luminance components of image patches with a size of 33×33 are obtained by randomly cropping the images to construct the training dataset Then, the PyTorch deep learning framework is used to implement the method proposed in the present invention. The model is trained for a total of 50 epochs, and the Adam optimizer is used to train the network. From the first epoch to the thirtieth epoch, the learning rate is set to 2e -4 ; from the 31st epoch to the 50th epoch, the learning rate is set to 1e -4 . The training image patch size is 33×33, the number of channels in the network convolutional layer is set to 32, the number of stages of iterative optimization is 9, the batch size is set to 64, and the size of the convolutional kernel is 3×3; finally, two metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), are used to evaluate the trained model.

[0028] Furthermore, in Step 2), the edge map f(x) is obtained by performing edge convolution on the original image x, that is,[[]] where E i ∈R w×w is the edge detection operator, and * is the convolution operation. The prior distribution of the edge map f(x) is P(f(x)) ∝ e -αg(f(x)) , and the implicit prior distribution of x guided by the edge map f(x) is P(x|f(x)) ∝ e -μΩ(xf(x)) , where g(f(x)) and Ω(x|f(x)) are energy functions related to x and f(x), and α and μ are two hyperparameters. Therefore, given the posterior value x of f(x) and the posterior values y and f(x) of x, it can be solved by Bayes' formula:

[0029]

[0030]

[0031] Among them, P(x) is the probability distribution of x and is independent of f(x). P(y|f(x)) is a marginal distribution of y and is independent of x. The maximum a posteriori (MAP) principle is used to maximize the log posterior probabilities P(f(x)|x) and P(x|y,f(x)) to obtain the final f(x) and x. Formulas (1) and (2) are transformed into the following optimization problems:

[0032]

[0033]

[0034] By performing a negative logarithm transformation, the above equation can be reformulated into an energy minimization model. Combining with the optimization formulas of image compressive sensing, Formulas (3) and (4) can be transformed into the following optimization problems:

[0035]

[0036]

[0037] Formulas (5) and (6) are the finally proposed optimization models for the edge map f(x) and the image x.

[0038] Furthermore, in step 3), since the data prior term has a non-differentiable property and cannot be directly solved by iteration, the proximal gradient descent algorithm is used to solve the optimization problem shown in Formula (5). First, the algorithm divides Formula (5) into two sub-problems, and uses the gradient descent algorithm and the proximal mapping algorithm to optimize these two problems respectively, and then iteratively optimizes them in sequence:

[0039] z (k) =f (k) (x (k-1) )-δ1(f (k) (x (k-1) )-Ex (k) ) (7)

[0040]

[0041] Among them, k represents the index value of iteration, δ1 represents a learnable step size parameter, prox α,g(·) is the proximal operator corresponding to the implicit prior g(f(x)). By alternately iteratively solving equations (7) and (8), the reconstruction solution of the edge map f(x) can be obtained. Then, the Half-Quadratic Splitting (HQS) algorithm is used to solve the optimization problem shown in equation (6). By introducing an auxiliary variable H, equation (6) can be reformulated as the following optimization problem:

[0042]

[0043] where μ1 is the penalty parameter. When μ1 approaches infinity, the solution of equation (9) converges to the solution of equation (6). Then, by alternately updating and optimizing x and H, the optimization problem of equation (9) is solved.

[0044] Update H: At the k-th iteration, there are the target image x (k) and the edge map f (k+1) (x (k) ). The auxiliary variable H can be updated as: [[ID=1)]]

[0045]

[0046] Using the proximal gradient descent algorithm to solve equation (10), we can obtain:

[0047] n (k) = H (k) - δ2(H (k) - x (k) ) (11)

[0048]

[0049] where δ2 is a learnable step size parameter. prox μ,Ω (·) is the proximal operator corresponding to the implicit prior Ω(Hf(x)).

[0050] Update x: Given H (k+1) , x can be updated as:

[0051]

[0052] Using the proximal gradient descent algorithm to solve equation (13), we can obtain:

[0053] r (k) = x (k) - δ3[Φ T (Φx (k) - y)- μ1(H (k+1) - x (k) )] (14)

[0054]

[0055] Among them, δ3 is a learnable step size parameter. T is the transpose operation of the matrix. prox λ,F (·) is the proximal operator corresponding to the implicit prior λ||F(x)||1.

[0056] Furthermore, in step 4), the sampling sub-network divides an image x ∈ R G×W into non-overlapping image patches of size Then each is reshaped into a new vector denotes the linear observation of the image patch, and the observation vector can be obtained through where Φ ∈ R is the sampling matrix, and the sampling sub-network uses convolutional operations to obtain the observation value M×N from

[0057] Furthermore, in step 5), the initial reconstruction sub-network uses as the initialization of the image patch in EGINet. Specifically, this sub-network consists of a 1×1 convolutional layer and a Pixel-Shuffle layer. First, the transpose matrix Φ T of Φ is obtained, and then it is reshaped into N filters of size 1×1×M. Then, using these filters, the 1×1 convolutional layer is used to obtain which is a tensor of size 1×1×N. Finally, the Pixel-Shuffle layer is used to reshape the tensor of size 1×1×N into an initialized image x of size In the EGINet network model, since H in the model is an approximation of the image patch x, H( (0) ) = x( 0 ) is used as the initialization of H. At the same time, the initialization of the edge map is obtained by convolving the initialized image of x with eight different gradients respectively, and then splicing the eight edge maps obtained by convolution along the channel dimension. 0 )

[0058] Furthermore, in step 6), the deep reconstruction sub-network expands formulas (8), (12), and (15) into the form of a network. In this algorithm, since the three proximal operators prox α,g (·), prox μ,Ω (·), and prox λ,F (·) have regularization terms and g(f(x)), Ω(Hf(x)), and F(x) cannot be explicitly derived, a deep convolutional neural network (DCNN) is used to learn and update x (k) ​, f (k) (x (k-1) ) and H (k) . The depth reconstruction sub-network consists of K stages, and each stage consists of an edge-aware feature extraction module (EAFM), an edge-guided intermediate variable update module (EGIM), and an intermediate variable-guided image reconstruction module (IGRM).

[0059] Edge-aware feature extraction module (EAFM): The network obtained by expanding equations (7) and (8) is called EAFM, and this module is mainly used to update the edge features of the image. First, the image x (k) and the edge map f (k) (x (k-1) ) are passed through the gradient descent algorithm shown in equation (7) to obtain an 8-channel feature z (k) . Secondly, for the implementation of the proximal mapping algorithm shown in equation (8), a 3×3 convolutional layer is used to increase the number of channels of the feature map to 32. Next, three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map. Then, three sets of repeated 3×3 convolutions and ReLU activation operations are used again to perform depth reconstruction on the edge features. Finally, a 3×3 convolution is used to change the 32-channel feature map into an 8-channel edge map for output. Residual connections are also introduced in the entire network to enhance the expressive power of the network.

[0060] Edge-guided intermediate variable update module (EGIM): The network obtained by expanding equations (11) and (12) is called EGIM, and this module is mainly used to guide the update of the intermediate variable H with the edge map f(x). First, the input image x (k) and the image H (k) are passed through the gradient descent algorithm shown in equation (11) to obtain a single-channel feature n (k) ; then, the edge map f (k+1) (x (k) ) obtained from the previous module and n (k)Perform a fusion operation. Here, a parallel cross-fusion module (PFM) is defined. This module first performs 3×3 convolution operations on the input edge features and image features respectively to extract features and increase the number of channels. Next, a set of obtained features and another set of features after Sigmod activation are multiplied crosswise to obtain complementary features, and then the convolved features are added. The obtained complementary features are selected by a gate mechanism to obtain the final fused features. Finally, the fused features are further enhanced by the proximal mapping algorithm shown in formula (12). This proximal mapping module first uses three sets of repeated 3×3 convolution and ReLU activation operations to extract the shallow features of the edge map. Then, it uses another three sets of repeated 3×3 convolution and ReLU activation operations to extract the depth features. Finally, a 3×3 convolution is used to convert the 32-channel feature map into a single-channel image for output. To reduce the information loss of the EGIM module between stages, a long short-term memory (LSTM) mechanism is also introduced to utilize the long and short-term units in the features to enhance the information representation ability. Among them, h and are the hidden states of different LSTMs, and c and are the cell states of different LSTMs.

[0061] Image reconstruction module guided by intermediate variables (IGRM): The network expanded from formula (14) and formula (15) is called IGRM. This module is mainly used to further reconstruct the final target image. This module takes the input image x (k) and the output image H of the EGIM module (k+1) to obtain a single-channel feature r (k) through the gradient descent algorithm shown in formula (14). Then, r (k) and the edge map f (k+1) (x (k) ) pass through the fusion module PFM to obtain the fused features. Finally, the fused features are used to deeply reconstruct the final target image through the proximal mapping algorithm shown in formula (15). In the proximal mapping module, first, three sets of repeated 3×3 convolution and ReLU activation operations are used to extract the shallow features of the edge map. Then, another three sets of repeated 3×3 convolution and ReLU activation operations are used to extract the depth features. Finally, a 3×3 convolution is used to convert the 32-channel feature map into a single-channel image for output. To reduce the information loss of the IGRM module between stages, a long short-term memory (LSTM) mechanism is also introduced to utilize the long and short-term units in the features to enhance the information representation ability. Among them, h and are the hidden states of different LSTMs, and c and are the cell states of different LSTMs.

[0062] Furthermore, in step 7), given the training dataset and Use the edge map f of the original image i and the reconstructed edge map f i (K) The error loss with L1 norm constraint between them is used as the loss function for predicting the edge map. Use the original image x i and the reconstructed image The error loss with L1 norm constraint between them is used as the loss function for image reconstruction, that is:

[0063]

[0064]

[0065] where N is the size of each image x i and N b is the number of training images, and K is the number of stages of EGINet. Therefore, the total loss function is:

[0066] L(Θ) = L rec + μL edge (18)

[0067] where are the learnable parameters set in EGINet, and μ is the regularization parameter. In subsequent experiments, μ is set to 0.1.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] 1. The present invention first proposes an interpretable compressive sensing image reconstruction model based on boundary constraints, then selects a suitable algorithm to solve the model, and finally unfolds the algorithm model into an edge-guided interpretable image compressive sensing reconstruction network (EGINet). Compared with other compressive sensing methods, the present invention has better interpretability.

[0070] 2. The deep reconstruction sub-network of the present invention has three sub-modules: an edge-aware feature extraction module (EAFM), an edge-guided intermediate variable update module (EGIM), and an intermediate variable-guided image reconstruction module (IGRM), which are respectively used to optimize the edge features, intermediate variable features, and image reconstruction features of the image. Since the present invention introduces the edge features of the image and incorporates them into the image reconstruction process, the reconstructed image has clearer edges and more detailed information.

[0071] 3. Generally, different features can be fused in an additive / concatenation along the channel dimension manner to enrich the information of the features. However, these methods cannot fully explore the relationship between the edge features and the image features, thus limiting the performance of image reconstruction. Therefore, in order to integrate the boundary information into the image reconstruction process, the present invention defines a parallel cross-fusion module (PFM), which can selectively fuse the boundary features and the image features. Therefore, compared with other fusion modules, this fusion module can well achieve the fusion between the edge map and the feature map. Description of the Drawings

[0072] Figure 1 It is a schematic diagram of the interpretable compressive sensing image reconstruction network based on boundary constraints of the present invention.

[0073] Figure 2 It is a schematic diagram of the edge map initialization network of the present invention.

[0074] Figure 3 It is a schematic diagram of the network of the k-th stage of the depth reconstruction sub-network of the present invention.

[0075] Figure 4 It is a schematic diagram of the networks of the BaseBlock module and the PFM module of the present invention.

[0076] Figure 5 It is a subjective quality comparison diagram of the method of the present invention and several recent interpretable compressive sensing methods for reconstructing the image named "Barbara" on the Set11 dataset.

[0077] Figure 6 It is a subjective quality comparison diagram of the method of the present invention and several recent interpretable compressive sensing methods for reconstructing the image named "216081" on the CBSD68 dataset.

[0078] Figure 7 It is a subjective quality comparison diagram of the method of the present invention and several recent interpretable compressive sensing methods for reconstructing the image named "img_054" on the Urban100 dataset. Detailed Implementation Manner

[0079] The present invention will be further described in detail below with reference to the drawings and embodiments. To better understand the image enhancement method of the present invention, the network structure of the present invention will be introduced in detail first.

[0080] I. Specific Implementation of the Interpretable Compressive Sensing Image Reconstruction Method Based on Boundary Constraints

[0081] As Figure 1As shown in Figure 1, the proposed boundary-constrained interpretable compressed sensing image reconstruction method has three subnetworks: sampling subnetwork, initial reconstruction subnetwork, and depth reconstruction subnetwork. The implementation steps of the proposed method are as follows:

[0082] Step 1) Construction of training and test datasets

[0083] Before network training, the T91 dataset is used as the training dataset of the network, and the Set11, CBSD68 and Urban100 datasets are used as the test datasets of the network.

[0084] Step 2) Sampling sub-network

[0085] The sampling subnetwork transforms an image x∈R G×W Divided into The size is Non-overlapping image patches Then each Reshape into a new vector Represents the linear observation of the image block, and the observation vector can be obtained by We get, where Φ∈R M×N is the sampling matrix. Figure 1 The sampling operation of image x is shown with a sampling rate of 25%. The sampling subnetwork uses convolution operations from Obtaining observations

[0086] Step 3) Initial reconstruction of the sub-network

[0087] Initial reconstruction of the subnet using As image blocks in EGINet Then, the Pixel-Shuffle layer is used to get the final initialization image x (0) Since H is an approximation of the image x, we use H (0) =x (0) As the initialization of H in EGINet. The edge graph f in EGINet (0) The initialization is obtained by convolving the initial image with 8 convolution kernels of different gradients, which can achieve feature fusion by performing splicing and convolution operations along the channel dimension, such as Figure 2 shown.

[0088] Step 4) Deep reconstruction sub-network

[0089] like Figure 3As shown, the depth reconstruction sub-network has a total of three modules: the edge-aware feature extraction module (EAFM), the edge-guided intermediate variable update module (EGIM), and the intermediate variable-guided image reconstruction module (IGRM), which are used to optimize the edge features, intermediate variable features, and image reconstruction features of the image, respectively.

[0090] Edge-aware feature extraction module (EAFM): This module is mainly used to update the edge features of the image. First, the image x (k) and the edge map f (k) (x (k-1) ) are used to obtain the 8-channel feature z (k) through the gradient descent algorithm. Then, a 3×3 convolutional layer is used to increase the number of channels of the feature map to 32. Next, three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map. Then, another three sets of repeated 3×3 convolutions and ReLU activation operations are used to perform depth reconstruction on the edge features. Finally, a 3×3 convolution is used to change the 32-channel feature map into an 8-channel edge map for output. Residual connections are also added to the entire network to enhance the information representation ability.

[0091] Edge-guided intermediate variable update module (EGIM): This module mainly uses the edge map f(x) to guide the update of the intermediate variable H. This module first takes the input image x (k) and the image H (k) to obtain the single-channel feature n (k) through the gradient descent algorithm. Then, the edge map f (k+1) (x (k) ) obtained from the previous module and n (k) are fused. Here, a parallel cross-fusion module (PFM) is defined. This module first performs 3×3 convolution operations on the input edge features and image features respectively to extract features and increase the number of channels. Then, a set of features obtained and another set of features after Sigmod activation are crossed to obtain complementary features. Then, the convolutional features are added, and the obtained complementary features are selected through a gate mechanism to obtain the final fused features. Then, three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map, and another three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the depth features. Finally, a 3×3 convolution is used to change the 32-channel feature map into a single-channel feature map for output. To reduce the information loss between stages of the EGIM module, a long short-term memory (LSTM) mechanism is also introduced to use the long and short-term units in the features to enhance the information representation ability. Among them, h and are the hidden states of different LSTMs, and c and are the cell states of different LSTMs.

[0092] Intermediate Variable Guided Image Reconstruction Module (IGRM): This module is mainly used to further reconstruct the final target image. This module takes the input image x (k) and the output image H of the EGIM module (k+1) to obtain a single-channel feature r through the gradient descent algorithm (k) , and then r (k) and the edge map f (k+1) (x (k) ) pass through the fusion module PFM to obtain the fused feature. Next, three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map, and three sets of repeated 3×3 convolutions and ReLU activation operations are used again to extract the deep features. Finally, a 3×3 convolution is used to convert the 32-channel feature map into a single-channel feature map for output. To reduce the information loss of the IGRM module between stages, a long short-term memory (LSTM) mechanism is also introduced to utilize the long and short-term units in the features to enhance the information representation ability, where h and are the hidden states of different LSTMs, and c and are the cell states of different LSTMs.

[0093] Step 5) Loss function

[0094] The loss function consists of the reconstruction error loss of the image and the reconstruction error loss of the edge map, and the L1 norm is used to constrain the reconstruction error loss.

[0095] II. Training of the Interpretable Compressed Sensing Image Reconstruction Method Based on Boundary Constraints

[0096] Network Training Parameter Configuration

[0097] First, randomly extract the luminance components of 88,912 image patches with a size of 33×33 to synthesize the training dataset That is, in EGINet, N = 1089 and N b = 88,912. Train EGINet sequentially in the case where the CS ratio ranges from {10%, 25%, 30%, 40%, 50%}. The experiments of the present invention are all carried out on a server device configured with an NVIDIA RTX A6000 GPU for training and testing. For the setting of network parameters, the training image patch size is 33, the number of channels is 32, the number of stages K is 9, the batch size is set to 64, the size of the convolution kernel is set to 3×3, and the model is trained for a total of 50 epochs. The optimizer used is the Adam optimizer. From the first Epoch to the thirtieth Epoch, the learning rate is set to 2e -4 ; from the 31st Epoch to the 50th Epoch, the learning rate is set to 1e-4 For fair testing, three widely used test datasets, including Set11, CBSD68, and Urban100, are used here. Two commonly used image evaluation criteria, namely peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are adopted to evaluate the model.

[0098] Network training

[0099] Given the training dataset and using the L1-norm constrained error loss between the edge map f of the original image i and the reconstructed edge map f i (K) as the loss function for predicting the edge map, and using the L1-norm constrained error loss between the original image x i and the reconstructed image as the loss function for image reconstruction, that is:

[0100]

[0101]

[0102] where N is the size of each image x i and N b is the number of training images, and K is the number of stages of EGINet. Therefore, the total loss function is:

[0103] L(Θ) = L rec + μL edge (18)

[0104] where are the learnable parameters set in EGINet, and μ is the regularization parameter. By default, μ is set to 0.1.

[0105] III. Testing of the interpretable compressive sensing image reconstruction method based on boundary constraints

[0106] 1. Comparison of objective indicators of network test results

[0107] To verify the feasibility of the method of this patent, it is compared with 4 representative non-interpretable image compressive sensing reconstruction methods, namely ReconNet, CSNet + , SCSNet, and DPA-Net, as shown in Table 1. Table 1 shows the comparison of the average PSNR / SSIM of several methods on the Set11 and CBSD68 datasets, where the best results are in bold and the second-best results are underlined. CSNet +SCSNet is trained using the BCDS500 training set, and ReconNet and DPA-Net are trained using the same dataset as the method of this patent. It can be observed from Table 1 that under the Set11 dataset, the average PSNR / SSIM of the method of this patent is better than the other four methods. Under the CBSD68 dataset, the SSIM of the method of this patent is better than the other four methods, but individual PSNR values are not the best. This is because the training datasets are different. The main contribution of the method of this patent is to add the edge information of the image, enabling the reconstructed image to better recover the edge features, that is, the SSIM value is better. Therefore, the method of this patent is better than these 4 representative unexplainable image compressive sensing reconstruction methods.

[0108] To verify the feasibility of the method of this patent, the method of this patent is compared with the latest 6 explainable compressive sensing methods, including ISTA-Net + 、OPINE-Net + 、MADUN, COAST, DGU-Net + and FSOINet. For a more fair comparison, this patent retrains and tests these 6 methods on a server device configured with an NVIDIA RTX A6000 GPU. Using the source code given by the author and training with the same training dataset as the method of this patent. Table 2 shows the comparison of the average PSNR / SSIM of several methods under five CS sampling rates on the Set11, CBSD68 and Urban100 datasets. It can be observed from Table 2 that under different sampling rates, the reconstruction performance of the method proposed in this patent is better than these explainable image compressive sensing reconstruction methods.

[0109] 2. Comparison of subjective indicators of network test results

[0110] To verify the effectiveness of the method of this patent, the explainable compressive sensing image reconstruction method based on boundary constraints proposed in this patent is compared with ISTA-Net + 、OPINE-Net + 、MADUN, COAST, DGU-Net + and FSOINet. Figure 5 、 Figure 6 and Figure 7 are the visual comparisons of several methods on the Set11, CBSD68 and Urban100 datasets respectively. It can be observed from these three figures that when CS = 30%, compared with other explainable image compressive sensing reconstruction methods, the image recovered by the method of this patent retains more detailed information, and at the same time the recovered edges are clearer, which fully verifies the superiority of the method of this patent.

[0111] Table 1

[0112]

[0113] Table 2

[0114]

[0115]

[0116] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claimed rights.

Claims

1. An interpretable compressive sensing image reconstruction method based on boundary constraints, characterized in that: It includes the following steps: Step 1) Construct the training data of the compressive sensing reconstruction model; Step 2) Construct the compressive sensing reconstruction model based on boundary constraints; First, use the edge convolution operator to convolve the image to obtain the edge map . Then, use Bayes' formula to represent the relationship between the image , the edge map and the observation . Finally, expand it into a compressive sensing reconstruction model with boundary constraints; Step 3) Optimize the compressive sensing reconstruction model with boundary constraints; Use the semi - quadratic splitting algorithm and the proximal gradient descent algorithm to optimize the compressive sensing reconstruction model based on boundary constraints obtained in Step 2), and expand it into an edge - guided interpretable image compressive sensing reconstruction networkEGINet; Step 4) Construct the sampling sub - network; Sample the input image using a sampling matrix to obtain the observations of the image ; Step 5) Construct the initial reconstruction sub - network; The observed values obtained in step 4) are deconvolved to achieve preliminary reconstruction of the image and obtain an initial reconstructed image . Then, from the initial reconstructed image an intermediate variable and an initial edge map are obtained; Step 6) Construct the deep reconstruction sub - network; Expand the optimized model obtained in Step 3) into a deep reconstruction sub - network, which has three modules: the edge - aware feature extraction module EAFM, the edge - guided intermediate variable update module EGIM, and the intermediate variable - guided image reconstruction module IGRM. These modules are used to optimize the edge features, intermediate variable features, and reconstruction features of the image respectively; The Edge-Aware Feature Extraction Module (EAFM) is used to update the edge features of an image. The image input to this module and the edge map are processed by the gradient descent algorithm to obtain 8-channel features ; The Edge-Guided Intermediate Variable Update Module (EGIM) uses the edge map to guide the update of the intermediate variable H. This module first processes the input image and the image by the gradient descent algorithm to obtain single-channel features , and then fuses the edge map and obtained from the previous module; The Intermediate Variable-Guided Image Reconstruction Module (IGRM) is used to further reconstruct the final target image. This module processes the input image and the output image of the EGIM module by the gradient descent algorithm to obtain single-channel features , and then and the edge map are fused through the Parallel Cross Fusion Module (PFM) to obtain fused features. Finally, the fused features are used as the input of the proximal mapping algorithm to reconstruct the final target image; Step 7) Loss function; The total objective loss consists of two parts: the reconstruction error loss of the image and the reconstruction error loss of the edge map. The L1 norm is used to constrain the reconstruction error loss.

2. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, wherein: The specific method for constructing the training data of the compressive sensing reconstruction model in Step 1) is as follows: First, the T91 dataset containing 91 images is used, and 88,912 luminance components of image patches with a size of 33×33 are obtained by randomly cropping the images to construct the training dataset , and the training of this model goes through a total of 50 epochs. The Adam optimizer is used to train the network. From the first epoch to the thirtieth epoch, the learning rate is set to 2e -4 ; From the 31st epoch to the 50th epoch, set the learning rate to 1e -4 , during the training of the EGINet network, the training image patch size is 33×33, the number of channels in the convolutional layer of the EGINet network is set to 32, the size of the convolutional kernel is set to 3×3, the number of iterative optimization stages of the EGINet network is 9, the batch size is set to 64. Finally, use two objective metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), to evaluate the trained model.

3. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, wherein: The edge map in step 2) is obtained by performing edge convolution on the image , that is,[[]] , where is an edge detection operator, and * is the convolution operation; Edge map has a prior distribution of , and the edge map guides the implicit prior distribution of , where and are energy functions related to and , and and are two hyperparameters. Therefore, given the posterior value of and the posterior value of as well as and the posterior value of and , it can be calculated by Bayes' formula:​​​​​​​ （1） （2） Among them, is 's probability distribution, independent of , is 's marginal distribution, independent of . Here, the maximum a posteriori probability estimation MAP formula is used to solve the posterior probability and , so as to obtain and . In other words, formula (1) and formula (2) can be transformed into the following optimization problem: （3） （4） By performing a negative logarithm transformation, formulas (1) and (2) are reformulated into an energy minimization model. Combining with the optimization formula of image compressive sensing, formulas (3) and (4) can be rewritten as: （5） （6） Formulas (5) and (6) are respectively the optimization models for the edge map and the image .

4. The method for reconstructing an interpretable compressive sensing image based on boundary constraints according to claim 1, wherein: The specific steps of Step 3) are as follows: First, use the proximal gradient descent algorithm to divide formula (5) into two sub - problems; then, use the gradient descent algorithm and the proximal mapping algorithm to optimize these two sub - problems respectively; finally, perform iterative optimization on them in sequence. The optimization solution formulas for these two sub - problems can be written as: （7） （8） wherein, represents the index value of iteration, is a learnable step size parameter, corresponds to the proximal operator of the implicit prior By alternately iteratively solving formulas (7) and (8), the reconstructed graph of the edge graph can be obtained. Then, the semi - quadratic splitting HQS algorithm is used to solve the optimization problem shown in formula (6). By introducing an auxiliary variable , formula (6) can be reformulated as the following optimization problem: （9） Among them, is the penalty parameter. When is close to infinity, the solution of formula (9) converges to the solution of formula (6). Then, by alternately updating and optimizing and , the optimization problem of formula (9) is solved. Update : Iteration When there is a target image and edge map , the auxiliary variable can be updated to: （10） Using the proximal gradient descent algorithm to solve formula (10) can obtain: （11） （12） Among them, is a learnable step size parameter, corresponds to the implicit prior proximal operator; Update : Given , can be updated to: （13） Using the proximal gradient descent method to solve formula (13) can obtain: （14） （15） where, is the updated step size, which is a learnable parameter, T is the transpose operation of the matrix, is the proximal operator corresponding to the implicit prior .

5. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, wherein: The sampling sub-network in step 4) divides an image into non-overlapping image patches of size , and then reshapes each into a new vector ∈ R N . ∈ R M represents the linear observation of the image patch, and the observation vector can be obtained by = Φ , where Φ ∈ R M×N is the sampling matrix, and the sampling sub-network uses convolution operations to obtain the observation value from .

6. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, wherein: In step 5), the initial reconstruction sub-network uses (0) = Φ T as the initialization of the image patch in the EGINet network. Specifically, this sub-network consists of a 1×1 convolutional layer and a Pixel-Shuffle layer. First, the transpose matrix of is obtained, and then it is reset to N filters of size 1×1×M. Then, using these filters, Φ is obtained through the 1×1 convolutional layer. Φ T is a tensor of size 1×1×N. Finally, the tensor of size 1×1×N is deformed into an initialization image of size using the Pixel-Shuffle layer; in the EGINet network model, since is an approximation of the image patch in Figure 1 , is used as the initialization of . At the same time, the initialization images of are convolved with eight different gradients respectively, and the eight edge maps obtained by convolution are concatenated along the channel dimension to obtain the initialized edge map.

7. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, characterized in that: In step 6), the depth reconstruction sub-network expands formulas (8), (12), and (15) into the form of a network; in this algorithm, since the three proximal operators , and have regularization terms and cannot be explicitly derived , and , a deep convolutional neural network (DCNN) is used to learn and update , and ; the depth reconstruction sub-network consists of K stages, and each stage consists of an edge-aware feature extraction module EAFM, an edge-guided intermediate variable update module EGIM, and an intermediate variable-guided image reconstruction module IGRM; Edge-Aware Feature Extraction Module EAFM: The network obtained by expanding Equations (7) and (8) is called EAFM, and this module is used to update the edge features of the image; first, the image input to this module and the edge map are processed by the gradient descent algorithm shown in Equation (7) to obtain 8-channel features . For the implementation of the proximal mapping algorithm shown in Equation (8), a 3×3 convolutional layer is used to increase the number of channels of the feature map to 32; then, three sets of repeated 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map; then, three sets of repeated 3×3 convolutions and ReLU activation operations are used again to perform deep reconstruction on the edge features; finally, a 3×3 convolution is used to change the 32-channel feature map into an 8-channel edge map for output; Edge-guided intermediate variable update module EGIM: The network expanded by formula (11) and formula (12) is called EGIM. This module is based on the edge graph To guide the update of the intermediate variable H; the module first inputs the image and images The single-channel feature is obtained by the gradient descent algorithm shown in formula (11) Then the edge map obtained by the previous module and To perform the fusion operation, a parallel cross fusion module PFM is defined here. The module first extracts features and increases the number of channels by performing 3×3 convolution operations on the input edge features and image features respectively. Then, the obtained set of features is cross-multiplied with another set of features after Sigmod activation to obtain complementary features. Then, the convolved features are added and the obtained complementary features are gated to obtain the final fusion features. Finally, the fused features are further enhanced by the proximal mapping algorithm shown in formula (12). In the proximal mapping module, three sets of repeated 3×3 convolution and ReLU activation operations are first used to extract the shallow features of the edge map. Then, three sets of repeated 3×3 convolution and ReLU activation operations are used again to extract the deep features. Finally, 3×3 convolution is used to convert the 32-channel feature map into a single-channel image for output. Intermediate Variable Guided Image Reconstruction Module IGRM: The network obtained by expanding Formula (14) and Formula (15) is called IGRM. This module is used to further reconstruct the final target image. This module takes the input image and the output image of the EGIM module to obtain single-channel features through the gradient descent algorithm shown in Formula (14) , and then and the edge map are fused through the parallel cross-fusion module PFM to obtain fused features. Finally, the fused features are used as the input of the proximal mapping algorithm shown in Formula (15), and the final target image can be reconstructed. In the proximal mapping module, first, three groups of 3×3 convolutions and ReLU activation operations are used to extract the shallow features of the edge map, then three groups of 3×3 convolutions and ReLU activation operations are used again to extract the deep features, and finally, a 3×3 convolution is used to change the 32-channel feature map into a single-channel image for output.

8. The method for reconstructing an interpretable compressed sensing image based on boundary constraints according to claim 1, wherein: In step 7), given a training data set and , using the edge map of the original image and the reconstructed edge map between The norm-constrained reconstruction error loss is used as the loss function of the edge map, using the original graph and reconstructed image between The norm-constrained reconstruction error loss is used as the loss function for image reconstruction, namely: （16） （17） where N is the size of each image and is the number of training images, K is the total number of stages of EGINet. Therefore, the total loss function can be written as: （18） Among them, are the learnable parameters of the EGINet network, is the regularization parameter. By default, is set to 0.1.

Images