Differential adaptive compressed sensing imaging device without reconstruction and method thereof

Abstract

The invention relates to a reconstruction-free differential self-adaptive compressed sensing imaging device and a reconstruction-free differential self-adaptive compressed sensing imaging method. The imaging device comprises a multi-stage sampling mode, and a first-stage sampling module adopts a uniform sampling mode, namely, each image block is sampled at the same sampling rate; the sampling modules of the second stage and the subsequent stages adopt a self-adaptive sampling mode, that is, each image block is sampled at different sampling rates, and a sampling strategy can be self-adaptively adjusted according to a current measured value during sampling; the measurement values obtained by the sampling modules are input into a scoring network, the scoring network outputs score values, and the sampling modules of the second level and the later levels conduct self-adaptive sampling on a target scene according to the score values; and after the sampling module finishes sampling, inputting all measurement values into the reconstruction network for reconstruction to obtain a final reconstruction result. According to the invention, in the multi-time adaptive sampling process, the distribution of the adaptive sampling rates of different image blocks can be completed without reconstruction.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and particularly relates to the field of adaptive compressive sensing imaging of images. Background Art

[0002] The theory of Compressive Sensing is a new sampling theory proposed in the 21st century. This theory proves that under the condition that the original signal satisfies sparsity, the original signal can be sampled at a rate significantly lower than the Nyquist sampling rate and the original signal can be almost accurately restored in the subsequent reconstruction process, thereby effectively reducing the amount of data at the acquisition end and transferring the computational load to the backend. Its mathematical model can be expressed as: y = φx, where x ∈ R N is an N-dimensional original signal, φ ∈ R M×N is an M×N-dimensional measurement matrix, y ∈ R M is an M-dimensional measurement result, where M << N, represents the sampling rate.

[0003] If the dimension of the original signal x is relatively high, it may lead to an overly large measurement matrix, thereby increasing the storage pressure. Therefore, in practical applications, the input image is often divided into blocks to relieve the storage pressure. Further, the sampling method can be divided into uniform sampling and adaptive sampling. The uniform sampling method samples each image block at the same sampling rate, which ignores the non-uniform spatial distribution of the image. Therefore, the adaptive sampling method is proposed to sample different image regions at different sampling rates according to the spatial distribution of the image. Adaptive sampling can further improve the sampling efficiency on the basis of the compressive sensing theory. The current adaptive compressive sensing imaging methods divide the sampling and reconstruction processes into multiple times. Before each sampling, the current measured values need to be reconstructed, and each sampling is adaptively adjusted according to the current reconstruction result. For example, by calculating the wavelet transform of the current low-resolution reconstruction result, the next sampling is only performed on the region with larger wavelet coefficients. However, since each sampling requires prior reconstruction and the reconstruction algorithm usually has a high complexity and long time consumption, this greatly increases the sampling time and makes it difficult for adaptive sampling to be applied to time-sensitive scenarios. Summary of the Invention

[0004] Aiming at the problems existing in the above-mentioned existing adaptive compressive sensing imaging methods, the object of the present invention is to propose a differentiable adaptive compressive sensing imaging device and method without reconstruction.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A differentiable adaptive compressive sensing imaging device without reconstruction, comprising a sampling module, a scoring network, and a reconstruction network. The imaging device includes multiple levels of sampling methods. Among them, the sampling module at the first level uses a uniform sampling method, that is, each image block is sampled at the same sampling rate; the sampling modules at the second level and each subsequent level use an adaptive sampling method, that is, each image block is sampled at different sampling rates, and the sampling strategy can be adaptively adjusted according to the current measurement value during sampling; the measurement values obtained by the sampling module are input into the scoring network, and the scoring network outputs a score value. The sampling modules at the second level and each subsequent level perform adaptive sampling on the target scene according to the score value; after the sampling module completes sampling, all the measurement values are input into the reconstruction network for reconstruction to obtain the final reconstruction result.

[0007] Further, the sampling modules at the second level and each subsequent level perform adaptive sampling on the target scene according to the score value, specifically: using a score-to-mask module to first convert the score value into a binary mask, and then selecting different rows of the measurement matrix according to the mask to perform adaptive sampling on different image blocks.

[0008] Further, the sampling module, the scoring network, and the reconstruction network are all differentiable.

[0009] Further, the dimension of the measurement matrix used by each level of sampling module is m k ×N, where m k is the number of sampling points for each image block in the k-th level of sampling, N is the data volume of each image block in the k-th level of sampling, m k <N, and the sampling rate achieved by each measurement matrix is

[0010] The present invention also provides a method using the above differentiable adaptive compressive sensing imaging device without reconstruction, including the following steps:

[0011] Step 1, the first-level sampling, which is uniform sampling, includes:

[0012] The original image is divided into blocks, and each image block is compressed and measured using the measurement matrix φ1 of the first level to obtain the measurement value y1 of the first level;

[0013] Step 2, the i-th level, which is adaptive sampling, 2 ≤ i ≤ k, includes:

[0014] Step 21, using the scoring network S1 to score the obtained measurement value y1 to obtain a score s1;

[0015] Step 22, according to the score s1, using the measurement matrix φ2 of the second level to perform adaptive sampling on the image divided in Step 1 to obtain the sampling result y2;

[0016] Step 23: Perform multi-level sampling according to the method in Steps 21-23 until the sampling at the k-th level is completed;

[0017] Step 3: Concatenate all the measurement values obtained in Step 1 and Step 2, and use the reconstruction network to reconstruct them to obtain the final reconstruction result

[0018] By designing a differentiable score-to-mask module and combining it with a score network, the present invention can realize end-to-end training of the entire adaptive compressive sensing imaging device. The beneficial effects of the method of the present invention are as follows:

[0019] (1) During multiple adaptive sampling processes, the allocation of adaptive sampling rates for different image patches can be completed without reconstruction;

[0020] (2) Each part of the entire adaptive compressive sensing imaging device is differentiable, enabling end-to-end training;

[0021] (3) The supervision of the reconstruction loss enables the optimization of each sampling allocation according to the reconstruction loss;

[0022] (4) Through the design of a multi-level sampling method, the sampling strategy can be adaptively adjusted according to the current measurement values during the second-level and subsequent level samplings, and compressive sensing imaging at different sampling rates can be achieved using one network. Description of the Drawings

[0023] Figure 1 is the flowchart of the method of the present invention;

[0024] Figure 2 is the specific structural schematic diagram of the score network in the embodiment of the present invention;

[0025] Figure 3 is the specific process schematic diagram of adaptive sampling according to the score map and the measurement matrix in the embodiment of the present invention;

[0026] Figure 4 is the specific structural schematic diagram of the reconstruction network in the embodiment of the present invention, where (a) is the backbone architecture of the reconstruction network, (b) is the structural schematic diagram of the downsampling module in the backbone architecture, (c) is the structural schematic diagram of the upsampling module in the backbone architecture, (d) is the schematic diagram of the DRB (Dense Residual Block) module in the backbone architecture, (e) is the schematic diagram of the RB (Residual Block) module in the DRB module, and (f) is the structural schematic diagram of the SA (Spatial Attention) module in the backbone architecture. Detailed Embodiment

[0027] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0028] The adaptive compressive sensing imaging device of this embodiment includes a sampling module, a scoring network, and a reconstruction network. Referring to Figure 1 , a differentiable adaptive compressive sensing imaging method without reconstruction in this embodiment is a multi-level sampling method, with a total of k levels. The input of the sampling module is the target scene to be reconstructed, and the output is the measurement value. The sampling module divides the target scene into image blocks with a spatial size of 8×8, and performs separate sampling on each image block. Among them, the sampling module in the first level adopts a uniform sampling method, that is, samples each image block at the same sampling rate; the sampling modules in the second level and each subsequent level adopt an adaptive sampling method, that is, sample each image block at different sampling rates, and can adaptively adjust the sampling strategy according to the current measurement value during sampling. To achieve adaptive sampling, first, the current measurement value needs to be input into the scoring network. The output of the scoring network is the score value. According to the score value, adaptive sampling is performed on the target scene to obtain the measurement value; after k samplings are completed, all the measurement values are input into the reconstruction network for reconstruction to obtain the final reconstruction result.

[0029] In this embodiment, a grayscale image is used as the input. When sampling, the original image is divided into image blocks with a spatial size of 8×8. The specific implementation includes the following steps:

[0030] Step 1, the first-level sampling, which is uniform sampling, includes:

[0031] Use the measurement matrix φ1, whose dimension is m1×N, where N = 8×8 = 64 is the data volume of each image block, and m1 < N. The sampling rate achieved by the measurement matrix φ1 is The dimension of the original grayscale image I is H×W×1, where H and W are the height and width of the image. Therefore, the number of sampling points for each 8×8 image block is m1. After sampling each image block separately, a compressed measurement result y1 with a dimension of can be obtained, and the corresponding sampling rate is The measurement matrix φ1 can be optimized end-to-end during the training process.

[0032] Step 2, the i-th level (2 ≤ i ≤ k) sampling, which is adaptive sampling, includes:

[0033] Use the scoring network to score the compressed measurement result obtained in the previous level to obtain the score value s i-1 . The scoring network is any convolutional neural network model that satisfies the input and output dimensions. The structure adopted in this embodiment is as Figure 2As shown, the scoring network includes two ConvNeXt Blocks. The specific structure of the first ConvNeXt Block is as shown in the expanded structure within the dashed box, and the second ConvNeXt Block has the same structure as the first one. The last layer of the scoring network is the SoftMax layer, which ensures that the sum of the output scores is 1.

[0034] Step 3, according to the score value s i-1 , use the measurement matrix φ i of the i-th level to perform adaptive sampling on the segmented image I to obtain the sampling result y i . The schematic diagram of the specific process of adaptive sampling according to the score map and the measurement matrix is as Figure 3 shown. Among them, the Score To Mask (STM) module first converts the score value s i-1 into a binary mask M i , where the number of "1"s in M i is the total number of sampling points m for the i-th adaptive sampling i , and the corresponding sampling rate for the i-th sampling is Then, the Score To Mask module selects different rows of the measurement matrix φ i to perform adaptive sampling on different image blocks according to the binary mask M i . Specifically, the positions of the "1" values in M i indicate the measurement values that need to be sampled. According to the positions of the "1" values, the corresponding rows of the measurement matrix are selected for sampling to obtain the measurement values.

[0035] During the training process, the Score To Mask module is non-differentiable. By introducing the auxiliary variable

[0036] η i = Concat[s i-1 ; s i-1 - 1 / m i ;...; s i-1 - (T i - 1) / m i ,

[0037] where T i is a hyperparameter that controls the maximum number of sampling points for each image block, Concat[.] is the data concatenation operation, s i-1 is the score value of the previous i - 1 samplings, and the obtained η i has a dimension of Therefore, in order to generate a binary mask M i with the number of "1"s being the total number of sampling points m for the i-th adaptive sampling i, the binarization calculation of the score-to-mask module can be expressed as:

[0038] M i = Binarize(η i - τ i ),

[0039] where τ i is the m i -th largest element in η i , and Binarize(.) is the binarization operation, where values greater than 0 are set to 1 and values less than 0 are set to 0. The above equation can be equivalently transformed into a linear programming problem, and its objective function and constraint space are:

[0040]

[0041]

[0042] where a, b, and t are the indices of the three dimensions of M i respectively.

[0043] This linear programming problem can be processed differentiably by introducing a perturbation optimizer. After introducing the perturbation optimizer, its forward propagation process can be expressed as:

[0044]

[0045] where Q different Gaussian noises Z q are added to η i , and the standard deviation of the noise is σ. By calculating the Jacobian matrix, the backpropagation process can be calculated, and its Jacobian matrix can be expressed as

[0046]

[0047] During the training process, in order to maintain the consistency between training and testing, the standard deviation σ of the added noise gradually decays to 0 as the number of iterations increases, and in order to prevent overfitting, the data in the third dimension of is randomly shuffled. During the testing process, no noise is added, and the data in the third dimension of is not randomly shuffled.

[0048] Step 4: Repeat Step 2-3 until the entire sampling process is completed to obtain the compressed measurement values at each level. During the multiple adaptive sampling processes, the next adaptive sampling scheme can be determined without reconstructing the image.

[0049] Step 5: Concatenate all the compressed measurement values and input them into the reconstruction network N. The reconstruction network N is any convolutional neural network model that satisfies the input and output dimensions. The specific structure of the reconstruction network N in this embodiment is asFigure 4 As shown. The first layer is a 1×1 convolutional layer, with an input dimension of and an output dimension of The second layer is a PixelShuffle layer, with an output dimension of H×W×32. The subsequent network is a parallel architecture with multi-scale and multi-branches, as shown in Figure 4 (a). In the longitudinal direction of the reconstruction network, there are upsampling and downsampling modules for generating feature maps of different scales, and their structures are respectively as shown in Figure 4 (b) and Figure 4 (c). The upsampling and downsampling modules contain a DRB (Dense Residual Block) module, and its structure is as shown in Figure 4 (d) and Figure (e). The reconstruction network in the transverse direction consists of SA modules (Spatial Attention), and the structure of the spatial attention module is as shown in Figure 4 (f). As shown in Figure 4 (a), there are 4 branches in the transverse direction. From top to bottom, the dimensions of the feature maps of each branch are H×W×32, and

[0050] In this embodiment, a step-by-step training method is adopted. Starting from the i-th (2≤i≤k) level, the existing sampling results (i.e., the concatenated measurement values) are input into the reconstruction network to output the reconstruction results. The network is trained by backpropagation using the reconstruction loss supervision. The reconstruction loss supervision function used in training the i-th level is the L1 loss between the final reconstruction result of the i-th level and the original image. This embodiment is trained for 5 levels, that is, k = 5, and the sampling rate for each level is set to 5%. During the training process of each level, it can be divided into two stages. In the first stage, all modules of the adaptive compressive sensing imaging device are trained for a total of 100 rounds. During this process, the noise σ is initially set to 0.05 and linearly decays to 0. And to prevent overfitting, the data of the third dimension is randomly shuffled . After that, the parameters of the scoring network are fixed. In the second stage of training, the perturbation optimizer is no longer used, and the data of the third dimension is no longer randomly shuffled . At this time, the training process is the same as the testing process. In the second stage, only the measurement matrix and the reconstruction network are trained for a total of 300 rounds, aiming to optimize the reconstruction quality. During the training process, it is necessary to fix all the previous parameters after training the previous level and continue to train the parameters of the next level until the training of the entire network is completed. The image patch size used in this implementation during training is 128×128, the optimizer is Adamoptimizer, and the learning rate is set to 2×10 -4 .

Claims

1. A differentiable adaptive compressive sensing imaging device without reconstruction, comprising a sampling module, a scoring network, and a reconstruction network, characterized in that, The imaging device includes a multi-level sampling method, where the sampling module at the first level uses a uniform sampling method, that is, samples each image block at the same sampling rate; the sampling modules at the second level and each subsequent level use an adaptive sampling method, that is, sample each image block at different sampling rates, and can adaptively adjust the sampling strategy according to the current measurement value during sampling; The measurement values obtained by the sampling module are input into the scoring network, and the scoring network outputs a score value. The sampling modules at the second level and each subsequent level perform adaptive sampling on the target scene according to the score value; After the sampling module completes sampling, all the measurement values are input into the reconstruction network for reconstruction to obtain the final reconstruction result.

2. The differentiable adaptive compressive sensing imaging device without reconstruction according to claim 1, wherein The sampling modules at the second level and each subsequent level perform adaptive sampling on the target scene according to the score value. Specifically: the score-to-mask module is used to first convert the score value into a binary mask, and then different rows of the measurement matrix are selected according to the mask to perform adaptive sampling on different image blocks.

3. The differentiable adaptive compressive sensing imaging device without reconstruction according to claim 2, characterized in that, The sampling module, the scoring network, and the reconstruction network are all differentiable.

4. A differentiable adaptive compressive sensing imaging device without reconstruction according to claim 2, characterized in that, The dimension of the measurement matrix used by each level of the sampling module is m k ×N, where m k is the number of sampling points for each image block in the k-th level of sampling, N is the data volume of each image block in the k-th level of sampling, and m k < N, and the sampling rate achieved by each measurement matrix is 5. A method of using the differentiable adaptive compressive sensing imaging device without reconstruction described in claim 1, characterized in that, This method includes the following steps: Step 1, first-level sampling, which is uniform sampling, including: The original image is block-processed, and each image block is compressed and measured using the measurement matrix φ1 at the first level to obtain the measurement value y1 at the first level; Step 2, the i-th level, which is adaptive sampling, 2 ≤ i ≤ k, including: Step 21, use the scoring network S1 to score the obtained measurement value y1 to obtain a score s1; Step 22, according to the score s1, use the measurement matrix φ2 at the second level to perform adaptive sampling on the image block-divided image in Step 1 to obtain a sampling result y2; Step 23, perform multi-level sampling according to the method in Steps 21 to 23 until the sampling at the k-th level is completed; Step 3, splice all the measurement values obtained in Step 1 and Step 2 together, and use the reconstruction network to reconstruct it to obtain the final reconstruction result.

6. The method according to claim 5, characterized in that The method of adaptive sampling in Step 22 is differentiable. The specific steps are: (1) use the score-to-mask module to convert the score s1 into a binary mask M1; (2) select different rows of the measurement matrix φ2 according to the binary mask M1 to perform adaptive sampling on different image blocks.

7. The method according to claim 6, characterized in that, The calculation process of the score-to-mask module converting the score into a binary mask is expressed as: M i = Binarize(η i - τ i ), Among them, η i is an auxiliary variable, τ i is the m i -th largest element in the auxiliary variable η i , and Binarize(.) is a binarization operation; η i = Concat[s i-1 ; s i-1 -1 / m i ; …; s i-1 -(T i -1) / m i ​ T i is a hyperparameter that controls the maximum number of sampling points for each image patch. Concat[.] is a data concatenation operation, and s i-1 is the sampling score value for the previous i - 1 times, and m i is the total number of points for the i-th adaptive sampling.

Images