A plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method

Abstract

The present invention relates to a plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method, which breaks through the constraints of the existing PnP method in which the reconstruction quality and reconstruction speed are mutually constrained and provides a new paradigm. First, in view of the special structure of the video SCI measurement matrix, a video SCI large-scale binary measurement matrix optimization method based on column coherence is proposed to improve the performance of the measurement matrix, thereby improving the quality of the reconstructed video. Then, a lightweight video denoising model based on a binary neural network is designed to convert the traditional floating-point convolution operation with high complexity into a 1-bit xnor operation and a bitcount counting mechanism, thereby reducing the computational complexity and the amount of parameters and improving the efficiency of the reconstruction algorithm. Compared with the existing PnP method, the present invention introduces measurement matrix optimization and model lightweighting, effectively reducing the reconstruction time on the basis of improving the reconstruction quality, and providing a new paradigm for reconstruction.

Description

Technical Field

[0001] The invention relates to the intersection of optical systems and computer vision, and in particular to a plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method. Background Art

[0002] Video Snapshot Compression Imaging (Video SCI) is an important application of compressed sensing technology in the field of optical systems and computer vision. It mainly consists of two parts: hardware encoding (the present invention selects the commonly used CACTI optical device) and reconstruction algorithm. First, Video SCI uses the CACTI optical device composed of digital micromirror devices (DMDs), lenses, CMOS / CCD and other optical components to modulate the incident light and compress the three-dimensional scene (such as video) into a two-dimensional image within a single exposure, which is the hardware encoding part; then based on the compressed sensing theory, a reconstruction algorithm based on iterative optimization or deep learning is used to restore the original three-dimensional scene, such as video. Video SCI uses the characteristics of compressed sensing technology to sample signals at a sampling standard far lower than the Nyquist-Shannon theorem, and can efficiently and low-costly collect, transmit, store and process the increasingly expanding massive image and video data, and has been widely used in aerospace, high-speed imaging, medical observation and other fields.

[0003] Among them, the plug-and-play video SCI reconstruction method directly uses a pre-trained deep denoiser for iterative optimization, which has the advantages of flexibility and high precision (that is, when the measurement matrix and the reconstruction object change, there is no need to retrain like the end-to-end deep learning model), and has great potential in the Video SCI reconstruction algorithm. However, limited by the performance and computational complexity of the deep denoiser, the reconstruction quality and reconstruction speed constrain each other. Specifically, a deep denoiser with good performance often requires a more complex network structure, resulting in a decrease in reconstruction speed; similarly, reducing the complexity of the deep denoiser network structure can increase the reconstruction speed, but it will sacrifice the reconstruction quality. Therefore, how to improve the quality of video SCI reconstruction without significantly decreasing or even increasing the reconstruction speed has always been the research focus and difficulty of the plug-and-play video SCI reconstruction method (hereinafter referred to as the PnP method). Summary of the invention

[0004] In view of the above problems, the present invention provides a plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method that improves reconstruction quality while reducing resource requirements and increasing reconstruction speed. The lightweight video denoising model in the present invention, BiFastDVDNet, is a lightweight video denoising model.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method, which specifically includes the following steps:

[0006] S1, using the video SCI large-scale binary measurement matrix optimization method based on column coherence, the optimized measurement matrix is ​​obtained

[0007] S2, build a lightweight video denoising model based on binary neural network And train, then save the trained weights;

[0008] S3, initialize the lightweight video denoising model based on binary neural network And load the weights trained in S2;

[0009] S4, the DMD in the CACTI optical device is loaded with the measurement matrix optimized in S1, and the measurement value y is obtained by using the SCI system forward model acquisition,

[0010] y = Φx + e (1.1) where x is the 2D image X of the first frame to the B frame k ,k∈[1,B] is vectorized and concatenated by column, Φ is the mask M of frame 1 to frame B respectively k ,k∈[1,B] is vectorized and concatenated row by row. The formula is as follows:

[0011]

[0012] S5, the reverse solution process of the SCI system is expressed as a regularization constrained optimization problem:

[0013]

[0014] Given v (k) and the optimized measurement matrix in S1 By v (k) On Linear Manifolds Euclidean projection update x (k+1) :

[0015]

[0016] S6, given x and noise level σ, using the Update v:

[0017]

[0018] S7, by continuously solving and iterating S5-S6 alternately, finally reconstructing the original video.

[0019] Preferably, the coherence is calculated as follows:

[0020]

[0021] The S1 specifically includes the following steps:

[0022] S101, data input,

[0023] Randomly initialize a Gaussian distributed mask As the initial weight, and then with two discrete cosine sparse basis Together as input, where N p Indicates the size of the patch;

[0024] S102, Binarization,

[0025] The mask M is input into the sign function and converted into a binary weight consisting of binary elements of 0 and 1. The calculation formula is as follows:

[0026]

[0027] S103, calculate the Gram matrices G and G′, the specific steps are as follows:

[0028] S103-1, binarize the kth frame Multiplying with the corresponding discrete cosine sparse basis Ψ yields the column-normalized perception matrix Θ k =norm(M k Ψ), and then use G k =Θ k T Θ k Calculate the Gram matrix G of different frames separately k , then the overall Gram matrix is ​​expressed as:

[0029] G=[G1,G2,...,G k ,...,G B ],k∈[1,B](1.8)

[0030] Where norm means column normalization;

[0031] S103-2, three-dimensional binary mask Input formula (1.2) and convert it into a two-dimensional binary measurement matrix Randomly crop a patch from it Then with the corresponding discrete cosine sparse basis Ψ p Multiply to get the column normalized perception matrix Final use Calculate the Gram matrix G′;

[0032] S104, calculate the loss, the formula is as follows:

[0033] L(M,Ψ)=argmin(L s +αL t +βL p +λL w )(1.9)

[0034] Where L s is the spatial coherence loss, L t is the time correlation loss, L p is the patch coherence loss, L w is the regularization loss, α, β and λ are correlation coefficients, and the specific steps are as follows:

[0035] S104-1, calculate the spatial coherence loss L s

[0036] The spatial coherence loss is represented by the maximum absolute value of the off-diagonal elements of the Gram matrix G in S103, that is:

[0037] μ(M k )=max 1≤i≠j≤n |g i,j |=||G k -I n || ∞ (1.10)

[0038] Then, summing the coherence of different frame masks, the spatial coherence L s Calculated by the following formula:

[0039]

[0040] in Indicates the mask M of the kth frame k Binarization of

[0041] S104-2, calculate the time correlation loss L t

[0042] L t Calculated by the following formula:

[0043]

[0044] M i and M j For any two frame masks, and Then they represent the binarized variables;

[0045] S104-3, calculate the patch coherence loss L p

[0046] The patch coherence is calculated by the maximum absolute value of the off-diagonal elements of the Gram matrix G′, that is:

[0047]

[0048] S104-4, calculate the regularized constraint loss L w

[0049] Perform L2 regularization on the floating-point mask M so that its value range is constrained between ±1:

[0050] L w =||M|| 2 (1.14);

[0051] S105, back propagation updates weight M

[0052] The sign function is approximated by a gradual two-stage approximate gradient method based on regularization constraints. The convergence is accelerated by the L2 regularization constraint of formula (1.14) based on EDE. The calculation is as follows:

[0053] g(M)=ktanhtM(1.15)

[0054]

[0055] Where N is the number of iterations, T min =10 -1 , T max =10 1 ;

[0056] S106, repeatedly iterate and optimize steps S102 to S105 until the loss value calculated by the loss function converges to obtain an optimized measurement matrix

[0057] S107, evaluating the optimization effect.

[0058] Preferably, S2 specifically includes the following steps:

[0059] S201, build a lightweight video denoising model based on binary neural network - BiFastDVDNet,

[0060] S201-1, divide the 5 frames of images into three groups, each with 3 frames, and input them into three BiDenoisigBlock1s for Stage 1 denoising respectively with the noise estimation, and output the image after the first denoising. The construction steps of each BiDenoisig Block1 are as follows;

[0061] S201-1-1, input, input three frames of images and noise levels into two Conv+BN+ReLU modules to extract features of images and noise levels;

[0062] S201-1-2, downsampling, input the features output in S201-1-1 into two downsampling modules to extract high-dimensional features. The module consists of 1 Conv+BN+ReLU, stride=2 and two BiConv+BN+RPReLU;

[0063] S201-1-3, upsampling, taking the high-dimensional features output by S201-1-2 as input, upsampling is performed through two upsampling modules consisting of BiConv+BN+RPReLU, Conv and Pixelshuffle;

[0064] S201-1-4, output the denoised image, convert the features output by S201-1-3 into an image through an image generation module composed of Conv+BN+ReLU and Conv, and output the image after the first denoising;

[0065] S201-2, input the first denoised images output by the three BiDenoisig Block1s into BiDenoisigBlock2 for Stage 2 denoising, repeat steps S201-1-1 to S201-1-4, and output the second denoised images;

[0066] S201-3, construct loss function

[0067] The loss function is the L2 Loss between the output image and the true value. The loss function calculation formula is as follows:

[0068]

[0069] in is the image output by the network, is a true value, It represents the mean square error of all floating-point convolution weights and 1;

[0070] S201-4, design back propagation function

[0071] Since the binary function sign is difficult to back-propagate, the binary convolution uses the sign approximation function AproxSign to perform a second-order fitting on the derivative of the sign function in the back-propagation to reduce the mismatch between the forward propagation and the reverse propagation. The AproxSign function and its gradient are shown in equations (2.8) and (2.9), respectively:

[0072]

[0073]

[0074] S202, build a data set and train the BiFastDVDnet model.

[0075] Preferably, the S202 specifically includes the following steps:

[0076] S202-1, randomly crop 20480000 videos with 96×96×5 spatiotemporal resolution from the training set of the DAVIS dataset to produce training samples, and use the DAVIS test set and Set8 dataset to test the performance of the model;

[0077] S202-2, randomly select 5 consecutive frames of images from the training sample {I n-2 ,I n-1 ,I n ,I n-1 ,I n+2}, and then input it together with a random noise level σ∈[5,55] into the lightweight video denoising model BiFastDVDnet built by S201;

[0078] S202-3, calculating the error using the loss function in S201-3;

[0079] S202-4, back propagation updates weights;

[0080] S202-5, iteratively optimize steps S202-2 to S202-4, train for 80 epochs in total, learn 256,000 samples in each epoch, and the learning rate gradually decreases with the increase of epochs. The learning rate for the first 50 epochs is 1e-3, the learning rate for 50 to 60 epochs is 1e-4, and the learning rate for the remaining epochs is 1e-6. Finally, save the trained model and weights as the lightweight denoising model in the reconstruction algorithm.

[0081] Preferably, the S105 specifically includes the following steps:

[0082] S105-1, when N=1, the derivative value of the gradient estimation function is kept close to 1;

[0083] S105-2, in the early iteration, the cutoff value is reduced from a large number to 1, and the back-propagation approximation function evolves from close to the Identity function to the Clip function;

[0084] S105-3, as the iteration continues, the derivative curve gradually evolves into the shape of a step function, and the approximate function of back propagation evolves from the Clip function to the sign function.

[0085] Preferably, the specific steps of S201-1-2 are as follows:

[0086] ①Distribution adjustment

[0087] Output feature x for Conv+BN+ReLU, stride=2 f The data distribution is adjusted to meet the binarization requirements of formula (2.2). The adjustment formula is as follows:

[0088] x f =kx f +b(2.1);

[0089] ② Input and weight binarization

[0090] The output x in ① f Binarization is performed as shown in formula (2.2), where x f Represents a full-precision number, x b Represents the binary value.

[0091]

[0092] Then the weight w in BiConv f Binarization is performed and a scaling factor α is introduced to increase the weight expression range and compensate for the information loss caused by binarization. As shown in equations (2.4) and (2.5), the optimal α is obtained by minimizing the quantization error. * and weight

[0093]

[0094]

[0095]

[0096] ③Xnor and bitcount operations

[0097] Use XOR and bit counting operations to replace floating-point multiplication and accumulation operations in convolutional neural networks, and perform the output x in ② b , Perform binary convolution, and the calculation formula is shown in formula (2.6):

[0098]

[0099] Where # represents xnor and bitcount operations, ⊙ represents the dot product with the scaling factor, x f and w f They are the full-precision input and weight before binarization in ②;

[0100] ④Output

[0101] Use batchnorm function to normalize the output in ③, then add it to the input, and finally output the calculation result through RPReLU activation function

[0102] Compared with the prior art, the present invention has the following beneficial effects:

[0103] 1. From the perspective of measurement matrix optimization, the present invention proposes a video SCI large-scale binary measurement matrix optimization method based on column coherence, and embeds it into the PnP method. It can effectively improve the reconstruction quality of the algorithm (0 to 3 dB above 8 compression) without increasing the network complexity of the deep denoiser (that is, without sacrificing the reconstruction speed).

[0104] 2. Compared with the traditional column coherence optimization method, the measurement matrix optimization method of the present invention has the following advantages on the basis of improving the performance of the measurement matrix:

[0105] ① Aiming at the special structure of the video SCI measurement matrix, that is, the matrix is ​​expanded diagonally and has a large scale, the present invention proposes a time-space-patch decomposition strategy, which effectively reduces the computational complexity of the large-scale measurement matrix, wherein the video memory and computational time are reduced by 10.7 times and 16.5 times respectively;

[0106] ② Aiming at the characteristic that the elements of the video SCI measurement matrix are composed of 0-1 binary discrete variables, the present invention solves the problem that binary discrete variables are difficult to optimize through a binary neural network, and proposes a progressive two-stage approximate gradient method (normEDE) based on regularization constraints to approximate the sign function and reduce the information loss in the gradient back propagation process;

[0107] 3. The present invention designs a lightweight video denoising model based on binary neural network, BiFastDVDNet, which quantizes the traditional floating-point convolution operation with high complexity into 1-bit xnor operation and bitcount counting mechanism, and reduces the computational complexity and parameter quantity by about half. By embedding it into the PnP method, the reconstruction speed can be effectively improved on the basis of only a slight decrease in reconstruction quality (taking the compression ratio of 8 as an example, the PSNR only decreases by 0.245dB);

[0108] 4. The present invention combines the measurement matrix optimization in point 1 and the model quantization method in point 3 into the PnP method, which can effectively improve the efficiency of the reconstruction algorithm while improving the reconstruction quality, and provide a new paradigm for breaking through the constraints of reconstruction quality and reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 is the reconstruction process of the present invention;

[0110] Figure 2 It is a binary measurement matrix optimization network of the present invention;

[0111] Figure 3 It is the measurement matrix verification method of the present invention;

[0112] Figure 4 It is the BiFastDVDNet video denoising network of the present invention;

[0113] Figure 5 It is the denoising binary convolution unit of the present invention

[0114] Figure 5 (a) is the model structure diagram of the denoising binary convolution unit. Figure 5 (b) is a schematic diagram of the forward propagation and back propagation functions of the denoising binary convolution unit;

[0115] Figure 6 It is the xnor operation and bitcount counting mechanism in the binary convolution module of the present invention, that is, Figure 5 A magnified image of BinaryConvolution in the middle;

[0116] Figure 7 It is the readjustment of the data distribution of the present invention

[0117] Figure 7 (a) is a diagram showing the data distribution of the ReLU activation function output before adjustment. Figure 7 (b) Schematic diagram of data distribution after readjustment of the output of the ReLU activation function. DETAILED DESCRIPTION

[0118] The following will be combined Figure 1-7 The present invention is described in detail. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0119] A plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method, specifically comprising the following steps:

[0120] The SCI system uses the principle that the modulation speed of the video frame (taking the F4320 DMD as an example, the maximum speed can reach 27995Hz) is faster than the capture rate of the camera, and uses the dynamic coded aperture (measurement matrix ), the 3D video frame sequence Compressed into 2D measurement frames The SCI system is expressed in the following mathematical form:

[0121] y = Φx + e (1.1) where x is the 2D image X of the first frame to the B frame k, k∈[1,B] is vectorized and then concatenated by column, that is, the two-dimensional images of different frames are expanded by column and then concatenated into a one-dimensional vector, Φ is the mask M of the first frame to the B frame respectively k ,k∈[1,B] is vectorized and concatenated row by row. The formula is as follows:

[0122]

[0123] The plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method is the reverse solution process of equation (1.1), that is, the process of restoring the original high-speed video frame x when y and Φ are known, which is expressed as a regularized constrained optimization problem:

[0124]

[0125] Where y and Φ are the measurement value and measurement matrix respectively, see formula (1.2) for details;

[0126] S1, such as Figure 2 As shown in the figure, the optimized measurement matrix is ​​obtained by using the video SCI large-scale binary measurement matrix optimization method based on column coherence

[0127] S2, build a lightweight video denoising model based on binary neural network And train, the network structure is as follows Figure 4 , Figure 5 and Figure 6 As shown, the trained weights are then saved;

[0128] S3, initialize the lightweight video denoising model based on binary neural network And load the weights trained in S2;

[0129] S4, the DMD in the CACTI optical device loads the measurement matrix optimized in S1 and acquires the measurement value y;

[0130] S5, given v (k) and the optimized measurement matrix in S1 By v (k) On Linear Manifolds Euclidean projection update x (k+1) :

[0131]

[0132] S6, given x and noise level σ, using the Update v:

[0133]

[0134] S7, by continuously solving and iterating S5-S6 alternately, finally reconstructing the original video.

[0135] The present invention mainly optimizes the measurement matrix and the deep denoiser (i.e., S1-S3) before executing steps S5 and S6, which can effectively reduce the computational complexity and the number of parameters while improving the quality of the reconstructed video, and provides a new paradigm for the PnP method to break through the constraints of the reconstruction quality and reconstruction speed and achieve efficient and high-quality reconstruction.

[0136] In order to retain more information during the measurement matrix encoding process and reconstruct higher quality video, coherence is usually used as an indicator to optimize the measurement matrix. The coherence is calculated as follows:

[0137]

[0138] From equations (1.1), (1.2) and (1.6), it can be found that, unlike image compression sensing, the video SCI measurement matrix and Gram matrix are larger in scale. Taking a 256×256×8 video as an example, the sizes of the measurement matrix and Gram matrix are [256×256, 256×256×8] and [256×256×8, 256×256×8], respectively, occupying more than 137.44GB of memory, making it difficult to optimize directly. Therefore, the present invention designs a video SCI large-scale binary measurement matrix optimization method based on column coherence, wherein step S1 specifically comprises the following steps:

[0139] S101, data input,

[0140] Randomly initialize a Gaussian distributed mask As the initial weight, and then with two discrete cosine sparse basis Together as input, where N p Indicates the size of the patch, see step S103-2 for details;

[0141] S102, Binarization,

[0142] The mask M is input into the sign function and converted into a binary weight consisting of binary elements of 0 and 1. The calculation formula is as follows:

[0143]

[0144] S103, calculate the Gram matrices G and G′, the specific steps are as follows:

[0145] S103-1, binarize the kth frame Multiplying with the corresponding discrete cosine sparse basis Ψ yields the column-normalized perception matrix Θ k =norm(M kΨ), and then use G k =Θ k T Θ k Calculate the Gram matrix G of different frames separately k , then the overall Gram matrix is ​​expressed as:

[0146] G=[G1,G2,...,G k ,...,G B ],k∈[1,B](1.8)

[0147] Where norm means column normalization;

[0148] S103-2, three-dimensional binary mask Input formula (1.2) and convert it into a two-dimensional binary measurement matrix Randomly crop a patch from it Then with the corresponding discrete cosine sparse basis Ψ p Multiply to get the column normalized perception matrix Final use Calculate the Gram matrix G′;

[0149] S104, calculate the loss. Based on the column coherence, the present invention decomposes the loss function of the three-dimensional measurement matrix optimization into four parts: two-dimensional spatial coherence (mask and sparse basis), two-dimensional temporal correlation (between different masks), patch coherence and regularization constraint, which can effectively reduce the calculation amount of large-scale measurement matrices. The calculation formula is as follows:

[0150] L(M,Ψ)=argmin(L s +αL t +βL p +λL w )(1.9)

[0151] Where L s is the spatial coherence loss, L t is the time correlation loss, L p is the patch coherence loss, L w is the regularization loss, α, β and λ are correlation coefficients, and the specific steps are as follows:

[0152] S104-1, calculate the spatial coherence loss L s

[0153] The spatial coherence loss is represented by the maximum absolute value of the off-diagonal elements of the Gram matrix G in S103, that is:

[0154] μ(M k )=max 1≤i≠j≤n |gi,j |=||G k -I n || ∞ (1.10)

[0155] Then, summing the coherence of different frame masks, the spatial coherence L s Calculated by the following formula:

[0156]

[0157] in Indicates the mask M of the kth frame k Binarization of

[0158] S104-2, calculate the time correlation loss L t

[0159] The present invention designs cosine similarity to constrain the similarity between masks of different frames. The lower the similarity, the lower the temporal correlation (i.e., the lower the correlation between masks of different frames). t Calculated by the following formula:

[0160]

[0161] M i and M j For any two frame masks, and Then they represent the binarized variables;

[0162] S104-3, calculate the patch coherence loss L p

[0163] The patch coherence (i.e., block coherence) is calculated by the maximum absolute value of the off-diagonal elements of the Gram matrix G′, that is:

[0164]

[0165] S104-4, calculate the regularized constraint loss L w

[0166] The floating-point mask M is constrained by L2 regularization to constrain its value range to ±1, reducing the information loss caused by binarization:

[0167] L w =||M|| 2 (1.14);

[0168] S105, back propagation updates weight M

[0169] Considering that the sign function is difficult to back propagate, the present invention approximates the sign function through a progressive two-stage approximate gradient method (normEDE) based on regularization constraints. On the basis of EDE, the L2 regularization constraint (see formula (1.14)) is used to make it converge faster. The calculation is as follows:

[0170] g(M)=ktanhtM(1.15)

[0171]

[0172] Where N is the number of iterations, T min =10 -1 , T max =10 1 , the specific steps are as follows:

[0173] S105-1, when N=1, the derivative value of the gradient estimation function is kept close to 1;

[0174] S105-2, in the early iteration, the cutoff value is reduced from a large number to 1, such as Figure 2 As shown in the lower left corner, the approximate function of back propagation evolves from close to the Identity function to the Clip function, thus ensuring the update capability in the early stage of training;

[0175] S105-3, as the iteration continues, the derivative curve gradually evolves into the shape of a step function, such as Figure 2 As shown in the lower right corner, the approximate function of back propagation evolves from the Clip function to the sign function, thus ensuring the consistency of forward and backward propagation.

[0176] S106, repeatedly iterate and optimize steps S102 to S105 until the loss value calculated by the loss function converges to obtain an optimized measurement matrix

[0177] S107, evaluate the optimization effect, such as Figure 3 As shown, the optimized mask M is divided into a series of 32×32×B patches. Then the coherence of all patches is calculated according to the formula in equation (1.6), and the maximum coherence is taken as the indicator for evaluating the effectiveness of the optimization method. The smaller the coherence, the better the optimization effect.

[0178] Considering the high computational complexity of full-precision (float32) convolutional neural networks, the present invention proposes a denoising binary convolution unit and embeds it into a deep denoiser, quantizes floating-point numbers into 1-bit binary numbers, and uses xnor operations and bitcount counting mechanisms to replace floating-point convolution operations, which can effectively reduce the number of parameters and computational complexity. The specific algorithm implementation steps of S2 include the following steps:

[0179] S201, build a lightweight video denoising model based on binary neural network - BiFastDVDNet, the specific network structure is as follows Figure 4 As shown in the upper part, the entire network is divided into two stages, namely Stage 1 and Stage 2 in the figure. Different stages share the same structure of BiDenoisig Block, such as Figure 4 The first stage consists of three BiDenoisig Blocks, which are named BiDenoisig Block1 for distinction; the second stage consists of one BiDenoisig Block, which is named BiDenoisig Block2 for distinction.

[0180] S201-1, divide the 5 frames of images into three groups, each with 3 frames, and input them into three BiDenoisigBlock1 (sharing the same structure and weight) for Stage 1 denoising respectively with the noise estimation, and output the image after the first denoising. The specific structure and construction steps of each BiDenoisigBlock1 are as follows;

[0181] S201-1-1, input, input three frames of images and noise levels into two Conv+BN+ReLU modules to extract features of images and noise levels;

[0182] S201-1-2, downsampling, input the features output in S201-1-1 into two downsampling modules to extract high-dimensional features. The module consists of 1 Conv+BN+ReLU, stride=2 and two BiConv+BN+RPReLU; BiConv+BN+RPReLU represents the invented denoising binary convolution unit, which quantizes the floating-point number into a 1-bit binary number through the sign function, and uses the xnor operation and bitcount counting mechanism to replace the floating-point convolution operation, which can effectively reduce the number of parameters and computational complexity. The specific steps are as follows:

[0183] ①Distribution adjustment

[0184] First, if Figure 7 As shown in (a), the features output by the ReLU activation function of the Conv+BN+ReLU, stride=2 part are all greater than 0, which makes it difficult to directly use them as the input of the denoising binary convolution unit (i.e., the BiConv+BN+RPReLU part) for binarization. f The data distribution is adjusted to meet the binarization requirements of formula (2.2). The adjustment formula is as follows:

[0185] xf =kx f +b(2.1);

[0186] like Figure 7 As shown in (b), the adjusted data range covers the positive and negative intervals, meeting the binarization requirements in formula (2.2); and most of the data is between positive and negative 1, which can reduce the loss of back propagation in formula (2.10) (data greater than 1 and less than -1 in back propagation will be truncated);

[0187] ② Input and weight binarization

[0188] The output x in ① f Binarization is performed as shown in formula (2.2), where x f Represents a full-precision number, x b Represents the binary value.

[0189]

[0190] Then the weight w in BiConv f Binarization is performed and a scaling factor α is introduced to increase the weight expression range and compensate for the information loss caused by binarization. As shown in equations (2.4) and (2.5), the optimal α is obtained by minimizing the quantization error. * and weight

[0191]

[0192]

[0193]

[0194] ③Xnor and bitcount operations

[0195] Use XOR and bit counting operations to replace floating-point multiplication and accumulation operations in convolutional neural networks (theoretically, the computational complexity is reduced by 64 times and the number of parameters is reduced by 32 times). b , Perform binary convolution, and the calculation formula is shown in formula (2.6):

[0196]

[0197] Where # represents xnor and bitcount operations, ⊙ represents the dot product with the scaling factor, x f and w f They are the full-precision input and weight before binarization in ②;

[0198] ④Output

[0199] The batchnorm function is used to normalize the output in ③, and then added to the input to effectively reduce the information loss caused by quantization. Finally, the calculation result is output through the RPReLU activation function.

[0200] S201-1-3, upsampling, taking the high-dimensional features output by S201-1-2 as input, upsampling is performed through two upsampling modules consisting of BiConv+BN+RPReLU, Conv and Pixelshuffle;

[0201] S201-1-4, output the denoised image, convert the features output by S201-1-3 into an image through an image generation module composed of Conv+BN+ReLU and Conv, and output the image after the first denoising;

[0202] S201-2, the output results of the three BiDenoisig Block1, that is, the image after the first denoising, are input into BiDenoisig Block2 (with the same structure as BiDenoisig Block1 but different parameters) for Stage 2 denoising. The construction and calculation steps are the same as those in S201-1, that is, steps S201-1-1 to S201-1-4 are repeated to output the image after the second denoising;

[0203] S201-3, construct loss function

[0204] The loss function is the L2 Loss between the output image and the true value. In addition, in order to reduce the quantization error, the present invention also adds a regularization loss to make the full-precision convolution weight before binarization close to 1, thereby reducing the information loss caused by binarization. The loss function calculation formula is as follows:

[0205]

[0206] in is the image output by the network, is a true value, It represents the mean square error of the sum of the squares of all floating-point convolution weights 1. The smaller the error, the closer the floating-point convolution weight is to ±1, and the smaller the information loss caused by the binary value.

[0207] S201-4, design back propagation function

[0208] Since the binary function sign is difficult to back-propagate, the binary convolution uses the sign approximation function AproxSign to perform a second-order fitting on the derivative of the sign function in the back-propagation to reduce the mismatch between the forward propagation and the reverse propagation. The AproxSign function and its gradient are shown in equations (2.8) and (2.9), respectively:

[0209]

[0210]

[0211] S202, build a data set and train the BiFastDVDnet model. The specific steps are as follows:

[0212] S202-1, randomly crop 20480000 videos with 96×96×5 spatiotemporal resolution from the training set of the DAVIS dataset to produce training samples, and use the DAVIS test set and Set8 dataset to test the performance of the model;

[0213] S202-2, randomly select 5 consecutive frames of images from the training sample {I n-2 ,I n-1 ,I n ,I n-1 ,I n+2}, and then input it together with a random noise level σ∈[5,55] into the lightweight video denoising model BiFastDVDnet built by S201;

[0214] S202-3, calculating the error using the loss function in S201-3;

[0215] S202-4, back propagation updates weights;

[0216] S202-5, iteratively optimize steps S202-2 to S202-4, train for 80 epochs in total, learn 256,000 samples in each epoch, and the learning rate gradually decreases with the increase of epochs. The learning rate for the first 50 epochs is 1e-3, the learning rate for 50 to 60 epochs is 1e-4, and the learning rate for the remaining epochs is 1e-6. Finally, save the trained model and weights as a lightweight denoiser in the reconstruction algorithm.

[0217] First, the present invention proposes a large-scale binary measurement matrix optimization method for video SCI based on column coherence to improve the performance of the measurement matrix. By embedding the optimized measurement matrix into the reconstruction algorithm, the reconstruction quality of video SCI can be improved by 0-3dB at a compression ratio of 8 or more. In addition, the time-space-patch decomposition method in the measurement matrix optimization method of the present invention can effectively reduce the demand for memory and computing resources of the video SCI measurement matrix (reduced by more than 10 times), which is particularly suitable for measurement matrix optimization in large-scale scenarios.

[0218] Secondly, the present invention also proposes a lightweight video denoising model based on a binary neural network, which quantizes the traditional floating-point convolution operation with high complexity into a 1-bit xnor operation and a bitcount counting mechanism, which can significantly reduce (by about half) the computational complexity (FLOPs) and the number of parameters while maintaining considerable accuracy (an average decrease of 0.245dB). Embedding it into the reconstruction method can effectively improve the efficiency of the reconstruction algorithm.

[0219] The technical solutions provided by the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the embodiments of the present invention. The description of the above embodiments is only applicable to help understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, according to the embodiments of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A plug-and-play, efficient, high-quality video snapshot compression imaging reconstruction method, characterized in that: The specific steps include: S1, using the video SCI large-scale binary measurement matrix optimization method based on column coherence, the optimized measurement matrix is ​​obtained S2, build a lightweight video denoising model based on binary neural network And train, then save the trained weights; S3, initialize the lightweight video denoising model based on binary neural network And load the weights trained in S2; S4, the DMD in the CACTI optical device loads the measurement matrix optimized in S1, and the measurement value y is obtained by using the SCI system forward model acquisition, y = Φx + e (1.1) where x is the 2D image X of the first frame to the B frame k ,k∈[1,B] is vectorized and concatenated by column, Φ is the mask M of frame 1 to frame B respectively k ,k∈[1,B] is vectorized and concatenated row by row. The formula is as follows: S5, the reverse solution process of the SCI system is expressed as a regularization constrained optimization problem: Given v (k) and the optimized measurement matrix in S1 By v (k) On Linear Manifolds Euclidean projection update x (k+1) : S6, given x and noise level σ, using the Update v: S7, by continuously solving and iterating S5-S6 alternately, finally reconstructing the original video.

2. The plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method according to claim 1, characterized in that: The coherence is calculated as follows: The S1 specifically includes the following steps: S101, data input, Randomly initialize a Gaussian distributed mask As the initial weight, and then with two discrete cosine sparse basis Together as input, where N p Indicates the size of the patch; S102, Binarization, The mask M is input into the sign function and converted into a binary weight consisting of binary elements of 0 and 1. The calculation formula is as follows: S103, calculate the Gram matrices G and G′, the specific steps are as follows: S103-1, binarize the kth frame Multiplying with the corresponding discrete cosine sparse basis Ψ yields the column-normalized perception matrix Θ k =norm(M k Ψ), and then use G k =Θ k T Θ k Calculate the Gram matrix G of different frames separately k , then the overall The Gram matrix is ​​represented as: G=[G1,G2,...,G k ,...,G B ],k∈[1,B](1.8)where norm represents column normalization; S103-2, three-dimensional binary mask Input formula (1.2) and convert it into a two-dimensional binary measurement matrix Randomly crop a patch from it Then with the corresponding discrete cosine sparse basis Ψ p Multiply to get the column normalized perception matrix Final use Calculate the Gram matrix G′; S104, calculate the loss, the formula is as follows: L(M,Ψ)=argmin(L s +αL t +βL p +λL w ) (1.9) Where L s is the spatial coherence loss, L t is the time correlation loss, L p is the patch coherence loss, L w is the regularization loss, α, β and λ are correlation coefficients, and the specific steps are as follows: S104-1, calculate the spatial coherence loss L s The spatial coherence loss is represented by the maximum absolute value of the off-diagonal elements of the Gram matrix G in S103, that is: μ(M k )=max 1≤i≠j≤n |g i,j |=||G k -I n || ∞ (1.10) Then, summing the coherence of different frame masks, the spatial coherence L s Calculated by the following formula: in Indicates the mask M of the kth frame k Binarization of S104-2, calculate the time correlation loss L t L t Calculated by the following formula: M i and M j For any two frame masks, and Then they represent the binarized variables; S104-3, calculate the patch coherence loss L p The patch coherence is calculated by the maximum absolute value of the off-diagonal elements of the Gram matrix G′, that is: S104-4, calculate the regularized constraint loss L w Perform L2 regularization on the floating-point mask M so that its value range is constrained between ±1: L w =||M|| 2 (1.14); S105, back propagation updates weight M The sign function is approximated by an asymptotic two-stage approximate gradient method based on regularization constraints. The convergence is accelerated by the L2 regularization constraint of formula (1.14) based on EDE. The calculation is as follows: g(M)=ktanhtM(1.15) Where N is the number of iterations, T min =10 -1 , T max =10 1 ; S106, repeatedly iterate and optimize steps S102 to S105 until the loss value calculated by the loss function converges to obtain an optimized measurement matrix S107, evaluating the optimization effect.

3. The plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method according to claim 1, characterized in that: The S2 specifically includes the following steps: S201, build a lightweight video denoising model based on binary neural network - BiFastDVDNet, S201-1, divide the 5 frames of images into three groups, each group has 3 frames, and respectively input the three BiDenoisig Block1s with noise estimation to perform Stage 1 denoising, and output the image after the first denoising. The construction steps of each BiDenoisig Block1 are as follows; S201-1-1, input, input three frames of images and noise levels into two Conv+BN+ReLU modules to extract features of images and noise levels; S201-1-2, downsampling, input the features output in S201-1-1 into two downsampling modules to extract high-dimensional features. The module consists of 1 Conv+BN+ReLU, stride=2 and two BiConv+BN+RPReLU; S201-1-3, upsampling, taking the high-dimensional features output by S201-1-2 as input, upsampling is performed through two upsampling modules consisting of BiConv+BN+RPReLU, Conv and Pixelshuffle; S201-1-4, output the denoised image, convert the features output by S201-1-3 into an image through an image generation module composed of Conv+BN+ReLU and Conv, and output the image after the first denoising; S201-2, input the first denoised images output by the three BiDenoisig Block1s into BiDenoisigBlock2 for Stage 2 denoising, repeat steps S201-1-1 to S201-1-4, and output the second denoised images; S201-3, construct loss function The loss function is the L2 Loss between the output image and the true value. The loss function calculation formula is as follows: in is the image output by the network, is a true value, It represents the mean square error of all floating-point convolution weights and 1; S201-4, design back propagation function Since the binary function sign is difficult to back-propagate, the binary convolution uses the sign approximation function AproxSign to perform a second-order fitting on the derivative of the sign function in the back-propagation to reduce the mismatch between the forward propagation and the reverse propagation. The AproxSign function and its gradient are shown in equations (2.8) and (2.9), respectively: S202, build a data set and train the BiFastDVDnet model.

4. The plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method according to claim 3, characterized in that: The S202 specifically includes the following steps: S202-1, randomly crop 20480000 videos with 96×96×5 spatiotemporal resolution from the training set of the DAVIS dataset to produce training samples, and use the DAVIS test set and Set8 dataset to test the performance of the model; S202-2, randomly select 5 consecutive frames of images from the training sample {I n-2 ,I n-1 ,I n ,I n-1 ,I n+2 }, and then input it together with a random noise level σ∈[5,55] into the lightweight video denoising model BiFastDVDnet built by S201; S202-3, calculating the error using the loss function in S201-3; S202-4, back propagation updates weights; S202-5, iteratively optimize steps S202-2 to S202-4, train for 80 epochs in total, learn 256,000 samples in each epoch, and the learning rate gradually decreases with the increase of epochs. The learning rate for the first 50 epochs is 1e-3, the learning rate for 50 to 60 epochs is 1e-4, and the learning rate for the remaining epochs is 1e-6. Finally, save the trained model and weights as the lightweight denoising model in the reconstruction algorithm.

5. The plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method according to claim 2, characterized in that: The S105 specifically includes the following steps: S105-1, when N=1, the derivative value of the gradient estimation function is kept close to 1; S105-2, in the early iteration, the cutoff value is reduced from a large number to 1, and the back-propagation approximation function evolves from close to the Identity function to the Clip function; S105-3, as the iteration continues, the derivative curve gradually evolves into the shape of a step function, and the approximate function of back propagation evolves from the Clip function to the sign function.

6. The plug-and-play high-efficiency and high-quality video snapshot compression imaging reconstruction method according to claim 3, characterized in that: The specific steps of S201-1-2 are as follows: ①Distribution adjustment Output feature x for Conv+BN+ReLU, stride=2 f The data distribution is adjusted to meet the binarization requirements of formula (2.2). The adjustment formula is as follows: x f =kx f +b (2.1); ② Input and weight binarization The output x in ① f Binarization is performed as shown in formula (2.2), where x f Represents a full-precision number, x b Represents the binary value. Then the weight w in BiConv f Binarization is performed and a scaling factor α is introduced to increase the weight expression range and compensate for the information loss caused by binarization. As shown in equations (2.4) and (2.5), the optimal α is obtained by minimizing the quantization error. * and weight w b* : ③Xnor and bitcount operations Use XOR and bit counting operations to replace floating-point multiplication and accumulation operations in convolutional neural networks, and perform the output x in ② b 、w b* Perform binary convolution, and the calculation formula is shown in formula (2.6): x f *w f ≈x b *w b* =(x b #w b )⊙α * (2.6) Where # represents xnor and bitcount operations, ⊙ represents the dot product with the scaling factor, x f and w f They are the full-precision input and weight before binarization in ②; ④Output The batchnorm function is used to normalize the output in ③, then added to the input, and finally the calculation result is output through the RPReLU activation function.

Images