Arbitrary scale adaptive compressed sensing reconstruction method based on convolutional neural network

Abstract

The present invention relates to an arbitrary scale adaptive compressed sensing reconstruction method based on a convolutional neural network. The method comprises the following steps: (1) using a linear fully convolutional neural network to perform compression measurement on an input image; (2) using a fully convolutional neural network to reconstruct the measurement result to obtain an initial reconstruction result; (3) inputting the reconstruction result into an attention module based on a convolutional neural network to obtain an attention map; (4) performing a dot multiplication of the attention map and the input image to obtain a modulated input image; (5) using a linear fully convolutional neural network to perform compression measurement on the modulated input image; (6) using a fully convolutional neural network to reconstruct the measurement result and add it to the previous reconstruction result to obtain a new reconstruction result; (7) repeating steps (3) to (6) to obtain a reconstruction result at a target compression rate. The present invention can process input images of arbitrary scales and achieve multiple sampling rates, thereby improving the robustness of the network and the efficiency of compression measurement.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular to the field of adaptive compressed sensing reconstruction for input images of arbitrary scales. Background Art

[0002] Compressed sensing theory is a new sampling theory proposed in the 21st century. It can sample the original signal at a rate far lower than the Nyquist sampling rate under the condition that the original signal meets the sparsity requirement and accurately restore the original signal, effectively reducing the amount of data at the acquisition end and transferring the computational load to the back-end. Its mathematical model is: y = ψx, where x∈R N , is the N-dimensional original signal, ψ∈R M×N is an M×N dimensional measurement matrix, y∈R M is the M-dimensional measurement result, where M<<N.

[0003] Typically, independent and identically distributed Gaussian random measurement matrices can serve as universal compressed sensing measurement matrices. However, if the original signal x is high-dimensional, the measurement matrix becomes excessively large, creating storage pressure. Therefore, in practice, the input image is often processed in blocks to reduce storage pressure, but this can introduce significant block artifacts. Because convolution can be expanded to represent traditional multiplication, with the recent development of deep learning, researchers have experimented with replacing traditional measurement matrices with linear fully convolutional neural networks. The advantage of using a fully convolutional network as the measurement matrix is ​​that it can process inputs of any scale without the need for image segmentation.

[0004] Adaptive compressed sensing is a way to improve sampling efficiency. Traditional adaptive compressed sensing uses manually designed masks to modulate the input image, essentially still a fixed-pattern measurement method. In deep learning, the attention mechanism was first proposed in natural language processing. Researchers later introduced it to image processing, achieving promising results. However, there is currently no method to introduce the attention mechanism into the field of compressed sensing.

[0005] After image compression measurement, the results need to be reconstructed. Traditional reconstruction algorithms face challenges such as high computational overhead, high complexity, and slow reconstruction speed. The advent of deep learning has significantly improved reconstruction efficiency and quality. However, existing reconstruction methods based on convolutional neural networks require training different networks for different sampling rates, significantly increasing algorithm complexity and reducing their practicality. Summary of the Invention

[0006] In view of the problems existing in the above existing compressed sensing reconstruction methods, the purpose of the present invention is to propose an adaptive compressed sensing reconstruction method for input images of any scale.

[0007] To achieve the above purpose, the calculation scheme adopted by the present invention is as follows:

[0008] An arbitrary scale adaptive compressed sensing reconstruction method based on a convolutional neural network includes the following steps:

[0009] Step 1: Use the linear fully convolutional neural network ψ1 to perform compression measurement on the input image I and obtain the measurement result y1;

[0010] Step 2: Use the fully convolutional neural network F1 to reconstruct the measurement result y1 and obtain the initial reconstruction result r1;

[0011] Step 3: Input the reconstruction result r1 into the attention module T1 based on the convolutional neural network to obtain the attention map A1;

[0012] Step 4: Perform a dot product of the attention map A1 and the input image I to obtain the modulated input image I1;

[0013] Step 5: Use the linear fully convolutional neural network ψ2 to perform compression measurement on the modulated input image I1 to obtain the measurement result y2;

[0014] Step 6: Use the fully convolutional neural network F2 to reconstruct the measurement result y2, and add the obtained result to the reconstruction result r1 in step 2 to obtain the reconstruction result r2;

[0015] Step 7: Repeat steps 3 to 6 until the reconstruction result r is obtained under the target compression rate. i , where i represents the number of iterations.

[0016] Furthermore, the specific structure of the linear fully convolutional neural network ψ1 in step 1 is:

[0017] The input is an m*n*1 image, and the output is The compression measurement results of the first convolution layer have 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is The second and third convolution layers each have s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The fourth and fifth convolutional layers have a total of s convolution kernels, the convolution kernel size is k*k, the step size is 2, and the pad is The sixth layer has 1 convolution kernel, the convolution kernel size is k*k, the step size is 2, and the pad is Among them, [] means rounding down.

[0018] Furthermore, the specific structure of the fully convolutional neural network F1 in step 2 is:

[0019] Input is The compression measurement results are output The first part of the reconstruction result is a convolution layer with s convolution kernels. The convolution kernel size is k*k, the step size is 1, and the pad is The second part is a residual dense block, where the convolution kernel size is k*k, the step size is 1, and the pad is The third part is three upsampling modules connected in series. The structure of each upsampling module is as follows: the first layer is a convolution layer with s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The second layer uses sub-pixel convolution as the upsampling layer, and the upsampling factor is set to 2. The third layer is the activation layer, and Leaky Relu is used as the activation function. The fourth part is a convolution layer with a total of 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is

[0020] Furthermore, the specific structure of the attention module T1 based on the convolutional neural network in step 3 is:

[0021] The first two layers are convolutional layers, each with a convolution kernel of size k*k, a step size of 1, and a pad size of The third layer is the activation layer, which uses the sigmoid function as the activation function.

[0022] To achieve adaptive compressed sensing reconstruction, the present invention uses a linear fully convolutional network instead of the traditional measurement matrix and introduces an attention module to modulate the input image, thereby achieving adaptive compressed sensing reconstruction that changes with the input. The beneficial effects of the present method are: (1) Because the entire network is a fully convolutional network, it can compress and reconstruct inputs of any scale without causing blockiness; (2) by introducing an attention mechanism to perform sampling and reconstruction multiple times, it is possible to use a single network to complete compressed sensing reconstruction at different sampling rates, and adaptive compressed sampling can improve the efficiency of compressed measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flow chart of the method of the present invention.

[0024] Figure 2 Schematic diagram of the specific structure of the compressed sampling network in an embodiment of the present invention.

[0025] Figure 3 Schematic diagram of the specific structure of the reconstruction network in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Reference Figure 1The embodiment of the present invention provides an arbitrary scale adaptive compressed sensing reconstruction method based on a convolutional neural network, which specifically includes the following steps:

[0028] Step 1, compression measurement: The compression measurement uses a linear fully convolutional neural network ψ1, whose input is an m*n*1 image I and the output is The compressed measurement result y1 of the first convolution layer has 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is The second and third convolution layers each have s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The fourth and fifth convolutional layers have a total of s convolution kernels, the convolution kernel size is k*k, the step size is 2, and the pad is The sixth layer has 1 convolution kernel, the convolution kernel size is k*k, the step size is 2, and the pad is It is worth noting that [] represents rounding down, and the convolution layer in ψ1 does not use any bias and activation function. This linear fully convolutional neural network can be equivalent to the measurement matrix in compressed sensing.

[0029] Step 2, reconstruction: The reconstruction process uses a fully convolutional neural network F1, whose input is The compression measurement result y1 is output The reconstruction result r1, the first part of which is a convolution layer, the convolution layer has s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The second part is a residual dense block, where the convolution kernel size is k*k, the step size is 1, and the pad is The third part is three upsampling modules connected in series. The structure of each upsampling module is as follows: the first layer is a convolution layer with s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The second layer uses sub-pixel convolution as the upsampling layer, and the upsampling factor is set to 2. The third layer is the activation layer, and Leaky Relu is used as the activation function. The fourth part is a convolution layer with a total of 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is

[0030] Step 3: Generate attention map: Use the attention module T1 based on convolutional neural network to generate attention map A1. The input of T1 is The reconstruction result r1 is output as The first two layers of attention map A1 are convolution layers, each with a convolution kernel of size k*k, a step size of 1, and a pad of The third layer is the activation layer, which uses the sigmoid function as the activation function.

[0031] Step 4, modulation: perform a dot product between the attention map A1 obtained in step 3 and the input image I to obtain the modulated input image I1.

[0032] Step 5: compress and measure again: Use the linear fully convolutional neural network ψ2 to compress and measure the modulated input image I1 to obtain the measurement result y2; ψ2 has the same structure as ψ1 in step 1, but the input objects are different, so the parameters cannot be shared.

[0033] Step 6: Reconstruct again: Use the fully convolutional neural network F2 to reconstruct the measurement result y2 and add it to the result r1 in step 2 to obtain the reconstructed result r2. It has the same structure as F1 in step 2, but does not share parameters.

[0034] Step 7: Repeat steps 3 to 6 until the reconstruction result r is obtained under the target compression rate. i , where i is determined by the target sampling rate. The sampling rate of each compression measurement is 1 / 64, and the target sampling rate is i times 1 / 64. It is worth noting that the network structure used in repeating steps 3 to 6 is the same as the network structure in steps 3 to 6, but the parameters are not shared. In step 7, the target compression rate is During training, the total compression rate is set to 0.5, which means a total of 32 samples are required. After completing the sampling twice in the above steps, steps 3 to 6 are repeated for 30 samples. During the training process, it is necessary to train the attention module, compressed sampling module, and reconstruction module in each iteration, fix the parameters of all previous iterations, and continue training the parameters of the next iteration until the training of the entire network is completed. After the entire network is trained, the reconstruction result of each iteration corresponds to a sampling rate that is an integer multiple of 1 / 64. The entire network can complete 32 different sampling rates, that is, sampling rates from 1 / 64 to 32 / 64.

Claims

1. An arbitrary scale adaptive compressed sensing reconstruction method based on convolutional neural network, characterized by: The following steps are involved: Step 1: Use the linear fully convolutional neural network ψ1 to perform compression measurement on the input image I and obtain the measurement result y1; the specific structure of the linear fully convolutional neural network ψ1 is: the input is an m*n*1 image, and the output is The compression measurement results of the first convolution layer have 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is The second and third convolution layers each have s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The fourth and fifth convolutional layers have a total of s convolution kernels, the convolution kernel size is k*k, the step size is 2, and the pad is The sixth layer has 1 convolution kernel, the convolution kernel size is k*k, the step size is 2, and the pad is Among them, [] means rounding down; Step 2: Use the fully convolutional neural network F1 to reconstruct the measurement result y1 and obtain the initial reconstruction result r1; Step 3: Input the reconstruction result r1 into the attention module T1 based on the convolutional neural network to obtain the attention map A1; the specific structure of the attention module T1 based on the convolutional neural network is as follows: the first two layers are convolution layers, each layer has 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is The third layer is the activation layer, which uses the sigmoid function as the activation function; Step 4: Perform a dot product of the attention map A1 and the input image I to obtain the modulated input image I1; Step 5: Use the linear fully convolutional neural network ψ2 to perform compression measurement on the modulated input image I1 to obtain the measurement result y2; The linear fully convolutional neural network ψ2 has the same structure as the network ψ1 in step 1, but does not share parameters; Step 6: Use the fully convolutional neural network F2 to reconstruct the measurement result y2, and add the obtained result to the reconstruction result r1 in step 2 to obtain the reconstruction result r2; Step 7: Repeat steps 3 to 6 until the reconstruction result r is obtained under the target compression rate. i , where i represents the number of iterations.

2. The arbitrary scale adaptive compressed sensing reconstruction method based on convolutional neural network according to claim 1 is characterized in that: The specific structure of the fully convolutional neural network F1 in step 2 is: Input is The compression measurement results are output The first part of the reconstruction result is a convolution layer with s convolution kernels. The convolution kernel size is k*k, the step size is 1, and the pad is The second part is a residual dense block, where the convolution kernel size is k*k, the step size is 1, and the pad is The third part is three upsampling modules connected in series. The structure of each upsampling module is as follows: the first layer is a convolution layer with s convolution kernels, the convolution kernel size is k*k, the step size is 1, and the pad is The second layer uses sub-pixel convolution as the upsampling layer, and the upsampling factor is set to 2. The third layer is the activation layer, and Leaky Relu is used as the activation function. The fourth part is a convolution layer with a total of 1 convolution kernel, the convolution kernel size is k*k, the step size is 1, and the pad is 3. The arbitrary scale adaptive compressed sensing reconstruction method based on convolutional neural network according to claim 1 or 2, characterized in that: The fully convolutional neural network F2 in step 6 has the same structure as F1 in step 2, but does not share parameters.

4. The arbitrary scale adaptive compressed sensing reconstruction method based on convolutional neural network according to claim 1 is characterized in that The target compression ratio in step 7 is An integer multiple of .

Images