Compressed sensing image restoration method coupling ConvLSTM and cross attention

Abstract

The invention discloses a compressed sensing image restoration method coupled with ConvLSTM and cross attention, and the method comprises the steps: constructing and training a deep expansion compressed sensing model coupled with ConvLSTM and a cross attention mechanism, and carrying out the restoration of an original image through employing the trained deep expansion compressed sensing model, and obtaining a reconstructed image. In each reconstruction stage, an adaptive gradient descent module AGDM and a near-end mapping module PMM are inserted on a cell path of the coupled ConvLSTM block, the coupled ConvLSTM block, the adaptive gradient descent module AGDM and the near-end mapping module PMM form a coupled LSTM attention module CLA, and after K reconstruction stages, convolution operation is performed again to output a final result. According to the method, the number of parameters of the model is reduced by using the coupled ConvLSTM, so that the method is more flexible and efficient, and degradation information of the image in multiple gradient descent iterations can be better captured.

Description

Technical Field

[0001] The present invention relates to image processing technology, and in particular to a compressed sensing image restoration method coupling ConvLSTM and cross attention. Background Art

[0002] Compressed sensing is a signal processing theory that breaks through the limitations of the traditional Nyquist sampling theorem and can reconstruct the original signal through a small amount of random sampling data at a sampling rate far lower than the Nyquist sampling rate. In the field of images, compressed sensing technology restores the complete image from some Fourier transform coefficients (K-space data) through sparse sampling and nonlinear reconstruction algorithms. In 2004, scientists such as Candes, Terence Tao and Donoho formally proposed the theory of compressed sensing, which attracted widespread attention from academia and industry as soon as it was proposed. With the deepening of research, the theory of compressed sensing has been widely used in information theory, image processing, earth science, optical / microwave imaging, pattern recognition, wireless communication, biomedical engineering and other fields, and was rated as one of the top ten scientific and technological advances in 2007 by American Science and Technology Review.

[0003] Traditional compressed sensing methods use the image sparsity of discrete cosine transform and wavelet as prior knowledge iteration, such as ISTA. Although these methods have significant convergence and strong interpretability, they have high computational complexity and difficulty in selecting the optimal parameters. With the rapid development of deep learning, these traditional methods have fallen far behind the image CS reconstruction methods based on deep learning. These methods can usually be divided into DUN (deep unfolded network) and DNUN (deep non-unfolded network).

[0004] However, DNUN is usually trained as a black box to directly learn the deep inverse mapping from the original domain to the measurement domain, so it has poor interpretability. DUN combines the excellent computing speed of deep learning with the interpretability based on traditional optimization, and has become the mainstream in the CS field, such as ISTA-NET+ and COSAT. However, deep unfolding networks based on traditional algorithms generally require multiple iterations (going through multiple gradient descent stages), which may cause serious degradation of the image and lose a lot of image information. In order to solve this problem, in recent years, some people have used ConvLSTM (convolutional long short-term memory mechanism) for deep unfolding networks based on traditional algorithms, such as MADUN and GCDUN. However, due to the uninterpretability of neural network calculations, ConvLSTM sometimes does not know what should be remembered and what should be forgotten, resulting in the loss of important image information. In addition, due to the existence of forget gates and input gates, ConvLSTM has complex calculations, slow training and convergence. In addition, the previous DUNS (deep unfolding network) has certain inherent defects. Once the step size ρ in their gradient descent module is completed, it will be fixed. However, different textures of the input image should be dynamically assigned corresponding step sizes, so many DUNS lack adaptability. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide a compressed sensing image restoration method coupling ConvLSTM and cross attention.

[0006] Technical solution: A compressed sensing image restoration method coupled with ConvLSTM and cross attention in the present invention builds and trains a deep unfolded compressed sensing model coupled with ConvLSTM and cross attention mechanism, and uses the trained deep unfolded compressed sensing model to restore the original image to obtain a reconstructed image x j ′ ;

[0007] The specific process of restoring the deep expansion compressed sensing model is as follows:

[0008] First, the sampling matrix Φ corresponding to the compressed sensing CS rate is used B Sparsely sample the input original image; then use the sampling matrix Φ B Transposed convolution Get the initialization image;

[0009] Then, the obtained initialization image is expanded and reconstructed for K times (e.g., 10 times), and in each reconstruction stage, a coupled LSTM attention module CLA is inserted on the cell path of the coupled ConvLSTM block, and each CLA module includes a gradient descent module AGDM and a proximal mapping module PMM;

[0010] Assume that x is the fully sampled original image, y is the image to be reconstructed after sampling by the sampling matrix, x (0) The initialization image is obtained by transposing convolution with kernel weight as sampling matrix, x (k) is the input of the K-th stage CLA module, h (k) is the hidden state of the K-th stage CLA module, then after K reconstruction stages, another convolution operation is performed to change the number of channels to 1, and the final output is x (k +1) The present invention combines the input gate and the forget gate of ConvLSTM into one, realizes parameter sharing, and reduces the complexity of the model.

[0011] Furthermore, the calculation process of the initialization image is:

[0012] Use the sampling matrix Φ corresponding to the compressed sensing CS rate B The input original image x is sparsely sampled; since each row of the sampling matrix can be regarded as a filter, the convolution operation is used to perform the sampling process. The sampling formula is as follows:

[0013] y j =Φ B x j ;

[0014] Then, using the sampling matrix Φ B Transposed convolution Get the initialization image y, the formula is:

[0015]

[0016] x j is the pixel value matrix of the jth sampling block in the original image, y j is the measurement value of the jth sampling block obtained after sampling.

[0017] Furthermore, for the Kth reconstruction stage, x (k) As the cell state that can be written and read by the gate, h contains short-term information (k) Fusion(x (k) and h (k) 4 times with different weights), and use the forget gate to adjust the input x (k) The product of the input gate (1-forget gate) supplements the key information of the cell path; to some extent, this supplement of information brings an inertial force to the gradient descent, making the convergence better;

[0018] The final output h of the Kth reconstruction stage (k) Output x by the latest proximal mapping module PMM (k)The sum of the output gate products (k -1) control;

[0019] To match the number of channels, in each CLA module, a convolution operation with a kernel size of 3 is used before and after the gradient descent module AGDM, and a convolution operation with a kernel size of 1 is used before the proximal mapping module PMM; at the same time, a convolution operation with a kernel size of 1 is used before each short-term memory h (k-1) and memory unit x (k-1) and short-term memory (k-1) and the output of AGDM (k) A residual operation is performed during integration, and the formula is as follows:

[0020] f (k) =σ(W xf *x (k-1) +W hf *h (k-1) +b f )

[0021] i (k) =1-f (k)

[0022]

[0023] s (k) =tanh(W xs *x (k-1) +W hs *h (k-1) +b s 0

[0024]

[0025] O (k) =σ(W xo *x (k-1) +W ho *h (k-1) +b o )

[0026]

[0027] Among them, * represents the paper operator, represents the hadama product, f (k) For the forget gate, i (k) is the input gate, whose weight is 1 minus f (k) The weight, r (k) 、s (k) Output of the intermediate process, o (k) is the output gate, h (k) is the hidden state of stage k, x (k)This is the final output of this stage. In the initialization stage, the hidden state h (0) Set to 0. In this case, the initial hidden state is 0, which allows the model to better focus on the input data x (0) rather than relying on an uncertain initial state.

[0028] Furthermore, the iterative step size of the gradient descent module AGDM adopts an adaptive generation step size P (k) , the specific method is:

[0029] First, the input of this stage Perform a convolution operation with a kernel size of 3, and then enter the normalized LN layer. For convenience, the input is abbreviated as z (k-1) , the specific formula is as follows:

[0030] a (k) =Conv 3x3 (LN(z (k-1) ));

[0031] where s (k) The input z of this stage (k-1) Output results after normalization and convolution layer;

[0032] Then, a (k) and z (k-1) Perform a cross-attention CA operation to adaptively generate the step size P from the degradation information and the current stage input (k) , the specific formula is as follows:

[0033] D (k) =Conv 3x3 (a (k) )

[0034] M (k) =RELU(Conv 3x3 (Conv 1x1 (D (k) *CA(a (k) ,h (k - 1) ))))

[0035] P (k) =M (k) +1

[0036] Among them, D (k) and M (k) Generate results for the intermediate;

[0037] r (k) =z (k-1) -P (k) Φ T (Φz (k-1)-y)

[0038] where z (k-1) is the reconstructed input value, y is the image after the sampling matrix is ​​sparse, P (k) is the step size in adaptive gradient descent, r (k) This is the final output result of adaptive gradient descent.

[0039] Furthermore, the proximal mapping module PMM processes the image r after being processed by the gradient descent module AGDM at the same stage. (k) For deep reconstruction, the proximal mapping module PMM includes a cross-attention mechanism to capture key information, multiple upsampling and multiple downsampling to perform multi-scale feature fusion. The specific contents are as follows:

[0040] First treat the reconstructed image r (k) Perform a series of convolution operations and gated fusion operations, and the output is R (k) Then the U-net network is used to reconstruct the image R (k) Perform two downsampling and two upsampling operations. Each downsampling reduces the length and width by half and expands the channel dimension to twice the previous one. Skip connections are deployed between levels to achieve cross-level feature fusion. In addition, a partial convolution operation is used before each upsampling and downsampling.

[0041] Perform two downsampling and two upsampling operations. Each downsampling reduces the length and width by half and expands the channel dimension to twice the previous one. Skip connections are deployed between levels to achieve cross-level feature fusion. In addition, a partial convolution operation is used before each upsampling and downsampling. In traditional convolution operations, occluded objects or missing information may have a negative impact on the convolution results, resulting in inaccurate processing results. Partial convolution reduces these negative impacts by ignoring invalid pixels, thereby improving the reliability of the processing results.

[0042] After downsampling, a cross-attention CSAB module is deployed, which has two inputs, namely the output image of the current K-stage downsampling and the downsampled output image of the same-level cascade in the K-1 stage;

[0043] After upsampling, a cross-attention CSAB module is deployed. This module has two inputs, namely the upsampled output image of the current K stage and the downsampled output image of the same cascade of K stages after a series of processing (channel merging with the upsampled image of the previous cascade and channel reduction through convolution with a kernel of 1 to match the number of channels of the current cascade);

[0044] The expression of the above process is as follows:

[0045]

[0046] r (k)↓ =DS(Pconv(CSAB(R (k) ,R (k - 1) )))

[0047] r (k)↓↓ =DS(Pconv(CSAB(r (k)↓ ,r (k - 1)↓ )))

[0048] r (k)↑ =US(Pconv(CSAB(r (k)↓↓ ,r (k - 1)↓↓ )))

[0049] r (k)↑↑ =US(Pconv(CSAB(Conv 1x1 (Concat(CSAB(r (k)↓ ,r (k - 1)↓ ),m (k) ))),r (k)↓ ))

[0050] q (k) =CSAB(Conv 1x1 (Concat(r (k)↑↑ ,CSAB(R (k) ,R (k - 1) ))))

[0051] x (k+1) =q (k) +R (k)

[0052] In the above formula, CSAB represents the cross attention module operation, Pconv represents the partial convolution operation, DS represents the downsampling operation, and US represents the upsampling operation; (k) is the output of adaptive gradient descent, i (k) is the input gate, s (k) ,q (k) is the intermediate result, x (k+1) It is the final output of this proximal mapping stage.

[0053] Furthermore, the above two figures are the cross-attention module of the present invention and the internal structure of the cross-attention mechanism. The cross-attention mechanism effectively captures the dependency between two sequences by calculating the similarity between two different sequences and assigning weights to each element according to the similarity;

[0054] In the present invention, during the downsampling process, the input received by the cross attention module comes from the same cascade of different stages. This operation effectively integrates the important information sequences of different stages, carries a large amount of memory meta-information of the previous stage, and gives greater weight to the important information of the two stages, so that the memory mechanism of the LSTM of the present invention makes timely and critical adjustments according to the weights of the important information of the two stages, selectively forgets the redundant information of the picture with small weight and some noise, so as to pay more attention to the image detail information with large weight;

[0055] The formula of the cross-attention module CSAB is as follows:

[0056] c (k) =LN(CSA(LN(r ′(k) ,r (k) )))

[0057] b (k) =Conv 1x1 (GELU(COnv 3x3 (GELU(COnv 1x1 (c (k) )))))

[0058] r″ (k) =c (k) +b (k)

[0059] where r′ (k) ,r (k) Represents the two inputs of the CSAB module in a cascade, GELU() represents the GELU activation function operation, Conv 1x1 () indicates that the kernel is 1 convolution layer, Conv 3x3 () indicates that the kernel is 3 convolution layers, r″ (k) is the final output; CSA() is the cross attention operation.

[0060] Furthermore, the training of the deep compressed sensing model was performed on a 3090 GPU, using a training dataset and a test dataset of 400 images from BSD500; the training images were cropped into 89,600 sub-images and data augmentation (rotation, flipping, etc.) was performed;

[0061] For network parameters, the block size is 32 and the default batch size is 16;

[0062] The network was trained using the Adam optimizer with an initial learning rate and a cosine annealing strategy for 80 epochs with an initial learning rate of 1x10 -4 , preheating epoach is 2;

[0063] For the loss function, the MSE loss function is used, and the formula is as follows:

[0064]

[0065] Among them, x j is the fully sampled real image, x j ′ is the image reconstructed by CS, N a and N represent the number of training images and the size of each image respectively.

[0066] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0067] (1) The present invention uses CoupledConvLSTM, which simplifies the internal structure of ConvLSTM by merging the input gate and the forget gate into a single "update gate". This simplification can reduce the number of parameters of the model and may help speed up the training process and reduce the risk of overfitting. Since the input gate and the forget gate are merged, the flow of information in the LSTM unit becomes more flexible and efficient, which helps to better capture the degradation information of the image in multiple gradient descent iterations.

[0068] (2) Using the cross-attention mechanism, a variant of the self-attention mechanism, the model focuses more on capturing key information and can dynamically pay attention to information at different locations in the image, thereby avoiding the LSTM mechanism from forgetting important information. It also prevents the LSTM from bringing harmful information into the next iterative CS reconstruction stage to a certain extent, greatly improving the model performance.

[0069] (3) The present invention uses the hidden unit h of the previous stage of LSTM (k-1) and the input x of this stage (k) Perform a cross-attention operation to adaptively generate the step size P from the degradation information and the current stage input (k) , with strong adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is the overall framework diagram of the present invention;

[0071] Figure 2 This is a schematic diagram of the Kth iterative reconstruction phase in the embodiment;

[0072] Figure 3 Schematic diagram of gradient descent module in the embodiment;

[0073] Figure 4 is a schematic diagram of a depth reconstruction module in an embodiment;

[0074] Figure 5Schematic diagram of the internal structure of the cross-attention module of the embodiment. DETAILED DESCRIPTION

[0075] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0076] In order to overcome the shortcomings of the prior art, the present invention combines the coupled ConvLSTM and Cross Attention, which is a variant of the self-attention mechanism, and applies them to the deep expansion network. The coupled ConvLSTM can greatly reduce the information loss caused by iterative expansion, and compared with ConvLSTM, the former combines the forget gate and the input gate into one, which solves the problem of slow ConvLSTM training and convergence speed. At the same time, the cross-attention mechanism enables the model to pay more attention to important information. The combination of cross-attention and coupled ConvLSTM not only utilizes the advantages of coupled ConvLSTM in processing long-term dependencies and training convergence speed, but also enhances the ability to capture key information through the cross-attention mechanism, greatly improving the accuracy of CS reconstruction. At the same time, based on the coupled ConvLSTM, the present invention uses the hidden state unit h of the previous stage as the cross-attention mapping, designs a content-adaptive gradient descent module, realizes image iterative adaptive step size update, and has strong adaptability.

[0077] like Figure 1 As shown, the overall model framework of the present invention is a deep expansion network for compressed sensing, which is expanded and iterated K times. In the present invention, K is set to 10. Each stage contains a CLA module, and each CLA contains an AGDM (adaptive gradient descent module) and a PMM (proximal mapping module). x is the original fully sampled image, y is the image to be reconstructed after sampling by the sampling matrix, x (0) is the image initialized by transposed convolution with kernel weight as sampling matrix, x (k) is the input of the K stage, h (k) is the hidden state of the K stage.

[0078] like Figure 2 As shown, the proximal gradient descent algorithm is coupled with the ConvLSTM block. AGDM and PMM are inserted into the cell path, as Figure 2 As shown, each reconstruction stage includes a GDM (gradient descent module) and a DPMM (deep reconstruction stage; therefore, x (k-1) is considered as a cell state that can be written and read by the gate. First, use x (k-1) as input and compare it with h which contains short-term information (k-1)Fusion. These two components are fused 4 times independently with different weights. The forget gate adjusts the input x (k-1) The product of the input gate (1-forget gate) supplements the key information of the cell path. To some extent, this supplement of information brings an inertial force to the gradient descent, making the convergence better. Output h (k) Output x by the latest PMM (Proximal Mapping Module) (k) The sum of the output gate products (k-1) To match the number of channels, a convolution operation with a kernel size of 3 was used before and after AGDM, and a convolution operation with a kernel size of 1 was used before PMM. (k-1) and memory unit x (k-1) and short-term memory (k-1) and the output of AGDM (k) A residual operation is performed during fusion to prevent unsuppressed harmful information from destroying the feature flow of the memory unit. The formula is as follows:

[0079] f (k) =σ(W xf *x (k-1) +W hf *h (k-1) +b f )

[0080] i (k) =1-f (k)

[0081]

[0082] s (k) =tanh(W xs *x (k-1) +W hs *h (k-1) +b s )

[0083]

[0084] o (k) =σ(W xo *x (k-1) +W ho *h (k-1) +b o )

[0085]

[0086] Among them, * represents the paper operator, Indicates the hadama product. (k) For the forget gate, i (k) is the input gate, whose weight is 1 minus f(k) The weight of r (k) ,s (k) Output of the intermediate process, o (k) is the output gate, h (k) is the hidden state of stage k, x (k) This is the final output of this stage. In the initialization stage, the hidden state h (0) Set to 0. In this case, the initial hidden state is 0, which allows the model to better focus on the input data x (0) rather than relying on an uncertain initial state.

[0087] like Figure 3 As shown in FIG, in the gradient descent algorithm, different textures in the image are assigned the same step size, so the existing algorithm lacks adaptability. In order to solve this problem, the present invention proposes a dynamic step size mapping module based on cross attention (such as Figure 3 First, the input of this stage Perform a convolution operation with a kernel size of 3, and then enter the normalized LN layer. For convenience, the input is abbreviated as z (k-1) , the specific formula is as follows:

[0088] a (k) =Conv 3x3 (LN(z (k-1) ));

[0089] where s (k) The input z of this stage (k-1) Output results after normalization and convolution layer;

[0090] Then, a (k) and z (k-1) Perform a cross-attention CA operation to adaptively generate the step size P from the degradation information and the current stage input (k) , the specific formula is as follows:

[0091] D (k) =Conv 3x3 (a (k) )

[0092] M (k) =RELU(Conv 3x3 (Conv 1x1 (D (k) *CA(a (k) ,h (k - 1) ))))

[0093] P (k) =M (k) +1.

[0094] That is to say, the present invention uses the hidden unit h^((k-1)) of the previous stage and the input x^((k)) of this stage to perform a cross-attention operation, thereby generating a step feature map, thereby achieving the effect of adaptively generating a gradient descent step.

[0095] like Figure 4 As shown, the proximal mapping module PMM processes the image r after being processed by the gradient descent module AGDM at the same stage. (k) For deep reconstruction, the proximal mapping module PMM includes a cross-attention mechanism to capture key information, multiple upsampling and multiple downsampling to perform multi-scale feature fusion. The specific contents are as follows:

[0096] First treat the reconstructed image r (k) Perform a series of convolution operations and gated fusion operations, and the output is R (k) Then the U-net network is used to reconstruct the image R (k) Perform two downsampling and two upsampling operations. Each downsampling reduces the length and width by half and expands the channel dimension to twice the previous one. Skip connections are deployed between levels to achieve cross-level feature fusion. In addition, a partial convolution operation is used before each upsampling and downsampling.

[0097] Perform two downsampling and two upsampling operations. Each downsampling reduces the length and width by half and expands the channel dimension to twice the previous one. Skip connections are deployed between levels to achieve cross-level feature fusion. In addition, a partial convolution operation is used before each upsampling and downsampling. In traditional convolution operations, occluded objects or missing information may have a negative impact on the convolution results, resulting in inaccurate processing results. Partial convolution reduces these negative impacts by ignoring invalid pixels, thereby improving the reliability of the processing results.

[0098] After downsampling, a cross-attention CSAB module is deployed, which has two inputs, namely the output image of the current K-stage downsampling and the downsampled output image of the same-level cascade in the K-1 stage;

[0099] After upsampling, a cross-attention CSAB module is deployed. This module has two inputs, namely the upsampled output image of the current K stage and the downsampled output image of the same cascade of K stages after a series of processing (channel merging with the upsampled image of the previous cascade and channel reduction through convolution with a kernel of 1 to match the number of channels of the current cascade);

[0100] The expression of the above process is as follows:

[0101]

[0102] r(k)↓ =DS(Pconv(CSAB(R (k) ,R (k-1) )))

[0103] r (k)↓↓ =DS(Pconv(CSAB(r (k)↓ ,r (k-1)↓ )))

[0104] r (k)↑ =US(Pconv(CSAB(r (k)↓↓ ,r (k-1)↓↓ )))

[0105] r (k)↑↑ =US(Pconv(CSAB(Conv 1x1 (Concat(CSAB(r (k)↓ ,r (k - 1)↓ ),m (k) ))),r (k)↓ ))

[0106] q (k) =CSAB(Conv 1x1 (Concat(r (k)↑↑ ,CSAB(R (k) ,R (k - 1) ))))

[0107] x (k+1) =q (k) +R (k)

[0108] In the above formula, CSAB represents the cross attention module operation, Pconv represents the partial convolution operation, DS represents the downsampling operation, and US represents the upsampling operation; (k) is the output of adaptive gradient descent, i (k) is the input gate, s (k) ,q (k) is the intermediate result, x (k+1) It is the final output of this proximal mapping stage.

[0109] like Figure 5 As shown, the above two figures are the cross attention module of the present invention and the internal structure of the cross attention mechanism. The cross attention mechanism effectively captures the dependency between two sequences by calculating the similarity between two different sequences and assigning weights to each element according to the similarity.

[0110] Note that in the present invention, during the downsampling process, the input received by the cross-attention module comes from the same cascade of different stages. This operation effectively integrates the important information sequences of different stages, carries a large amount of memory meta-information from the previous stage, and gives greater weight to the important information of the two stages, so that the LSTM memory mechanism of the present invention makes timely and critical adjustments based on the weights of the important information of the two stages, selectively forgets the redundant information of the image with small weight and some noise, so as to pay more attention to the image detail information with large weight. The specific formula of the overall module is as follows:

[0111] c (k) =LN(CSA(LN(r ′(k) ,r (k) )))#(twenty two)

[0112] b (k) =Conv 1x1 (GELU(Conv 3x3 (GELU(Conv 1x1 (c (k) )))))#(twenty three)

[0113] r″ (k) =c (k) +b (k) #(twenty four)

[0114] where r′ (k) ,r (k) Represents the two inputs of the CSAB module in a cascade, GELU() represents the GELU activation function operation, Conv 1x1 () indicates that the kernel is 1 convolution layer, Conv 3x3 () indicates that the kernel is 3 convolution layers, r″ (k) is the final output. CSA() is the cross attention operation.

Claims

1. A compressed sensing image restoration method coupling ConvLSTM and cross attention, characterized in that: Construct and train a deep unfolding compressed sensing model that couples ConvLSTM and cross attention mechanism, and use the trained deep unfolding compressed sensing model to restore the original image to obtain the reconstructed image x j ′ ; The specific process of restoring the deep expansion compressed sensing model is as follows: First, the sampling matrix Φ corresponding to the compressed sensing CS rate is used B Sparsely sample the input original image; then use the sampling matrix Φ B Transposed convolution Get the initialization image; Then, the obtained initialization image is expanded and reconstructed for K iterations. In each reconstruction stage, an adaptive gradient descent module AGDM and a proximal mapping module PMM are inserted into the cell path of the coupled ConvLSTM block. The coupled ConvLSTM block, the adaptive gradient descent module AGDM and the proximal mapping module PMM form a coupled LSTM attention module CLA. Assume that x is the fully sampled original image, y is the image to be reconstructed after sampling by the sampling matrix, x (0) The initialization image is obtained by transposing convolution with kernel weight as sampling matrix, x (k) is the input of the K-th stage CLA module, h (k) is the hidden state of the K-th stage CLA module, then after K reconstruction stages, another convolution operation is performed to change the number of channels to 1, and the final output is x (k+1) .

2. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 1, characterized in that: The calculation process of the initialization image is: Use the sampling matrix Φ corresponding to the compressed sensing CS rate B The input original image x is sparsely sampled; since each row of the sampling matrix can be regarded as a filter, the convolution operation is used to perform the sampling process. The sampling formula is as follows: y j =Φ B x j ; Then, using the sampling matrix Φ B Transposed convolution Get the initialization image y, the formula is: x j is the pixel value matrix of the jth sampling block in the original image, y j is the measurement value of the jth sampling block obtained after sampling.

3. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 1, characterized in that: For the Kth reconstruction phase, x (k) As the cell state that can be written and read by the gate, h contains short-term information (k) Fusion, use forget gate to adjust input x (k) The product of the input gate supplements the key information of the cell pathway; The final output h( k ) is output by the latest proximal mapping module PMM ( k ) and the output gate product o (k-1) control; To match the number of channels, each coupled LSTM attention module CLA uses a convolution operation with a kernel size of 3 before and after the gradient descent module AGDM, and a convolution operation with a kernel size of 1 before the proximal mapping module PMM; at the same time, each short-term memory h (k-1) and memory unit x (k-1) and short-term memory h( k - 1 ) and the output r( k ) is merged to perform a residual operation, the formula is as follows: f (k) =σ(W xf *x (k-1) +W hf *h (k-1) +b f ) i (k) =1-f (k) s (k) =tanh(W xs *x (k-1) +W hs *h (k-1) +b s ) o (k) =σ(W xo *x (k-1) +W ho *h (k-1) +b o ) Among them, * represents the paper operator, represents the Haddam product, f(k) is the forget gate, i(k) is the input gate, and its weight is 1 minus f( k ) weight, r( k ) is the output of the adaptive gradient descent module, s( k ) is the intermediate process output, o (k) is the output gate, h (k) is the hidden state of stage k, x (k) This is the final output of this stage.

4. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 1 or 3, characterized in that: The iterative step size of the gradient descent module AGDM adopts the adaptive generation step size P (k) , the specific method is: First, the input of this stage Perform a convolution operation with a kernel size of 3, and then enter the normalized LN layer. For convenience, the input is abbreviated as z (k-l) , the specific formula is as follows: a (k) =Conv 3x3 (LN(z (k-1) )); where s (k) The input z of this stage (k-1) Output results after normalization and convolution layer; Then, a( k ) and z( k - 1 ) performs a cross-attention CA operation to adaptively generate the step size P from the degradation information and the current stage input (k) , the specific formula is as follows: D (k) =Conv 3x3 (a (k) ) M (k) =RELU(Conv 3x3 (Conv. 1x1 (D (k) *CA(a (k) ,h (k-1) )))) P (k) =M (k) +1 Among them, D( k ) and M( k ) is the intermediate generated result; r (k) =z (k-1) -P (k) Φ T (Φz (k-1) -y) where z (k-1) is the reconstructed input value, y is the image after the sampling matrix is ​​sparse, P (k) is the step size in adaptive gradient descent, r( k ) is the final output result of adaptive gradient descent.

5. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 1 or 3, characterized in that: The proximal mapping module PMM processes the image r( k ) for deep reconstruction, the specific contents are as follows: First treat the reconstructed image r (k) Perform a series of convolution operations and gated fusion operations, and the output is R (k) Then use U- net The network treats the reconstructed image R( k ) performs two downsampling and two upsampling operations, each downsampling reduces the length and width by half and expands the channel dimension to twice the previous one, and deploys skip connections between levels to achieve cross-level feature fusion; and a partial convolution operation is used before each upsampling and downsampling; After two downsamplings, a cross-attention CSAB module is deployed, which has two inputs, namely the output image of the current K-stage downsampling and the downsampled output image of the same-level cascade in the K-1 stage; After two upsamplings, a cross-attention CSAB module is deployed. This module has two inputs, namely the upsampled output image of the current K stage and the downsampled output image of the same cascade after a series of processing in the K stage. The expression of the above process is as follows: r (k)↓ =DS(Pconv(CSAB(R (k) ,R (k-1) ))) r ck)↓↓ =DS(Pconv(CSAB(r (k)↓ ,r (k-1)↓ ))) r (k)↑ =US(Pconv(CSAB(r (k)↓↓ ,r (k-1)↓↓ ))) r (k)↑↑ =US(Pconv(CSAB(Conv 1x1 (Concat(CSAB(r (k)↓ ,r(k-1) ↓ ),m (k) ))),r (k)↓ )) q (k) =CSAB(Conv 1x1 (Concat(r (k)↑↑ ,CSAB(R (k) ,R (k-1) )))) x (k+1) =q (k) +R (k) In the above formula, CSAB represents the cross attention module operation, Pcon v represents a partial convolution operation, DS represents a downsampling operation, and US represents an upsampling operation; r( k ) is the output of adaptive gradient descent, i( k ) is the input gate, s (k) ,q (k) is the intermediate result, x (k +1) It is the final output of this proximal mapping stage.

6. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 5, characterized in that: The formula of the cross-attention module CSAB is as follows: c (k) =LN(CSA(LN(r′ (k) ,r (k) ))) b (k) =Conv 1x1 (GELU(Conv 3x3 (GELU(Conv 1x1 (c) (k) ))))) r″ (k) =c (k) +b (k) where r′ (k) , r (k) Represents the two inputs of the CSAB module in a cascade, GELU() represents the GELU activation function operation, Conv 1x1 () indicates that the kernel is 1 convolution layer, Conv 3x3 () indicates that the kernel is 3 convolution layers, r″ (k) is the final output; CSA() is the cross attention operation.

7. The compressed sensing image restoration method coupled with ConvLSTM and cross attention according to claim 1, characterized in that: When training the deep compressed sensing model, the input image is cropped into 89,600 sub-images and data augmentation is performed; For network parameters, the block size is 32 and the default batch size is 16; The network was trained using the Adam optimizer with an initial learning rate and a cosine annealing strategy for 80 epochs with an initial learning rate of 1x10 -4 , preheating epoach is 2; Using the MSE loss function, the formula is as follows: Among them, x j is the fully sampled real image, x j ′ is the image reconstructed by CS, N a and N represent the number of training images and the size of each image respectively.

Images