A high-efficiency distributed video compressive sensing reconstruction system and method

Abstract

The application relates to a high-efficiency distributed video compressed sensing reconstruction system and method, and belongs to the technical field of image processing. ++ The ISTA-Net is introduced into the distributed video compressed sensing system to independently reconstruct and initially reconstruct key frames and non-key frames, and the intra-frame correlation is fully utilized. Through the designed two-stage deep reconstruction network module, the inter-frame correlation is fully utilized to deeply reconstruct the non-key frames. The deep reconstruction network module FMC-Net of the first stage carries out motion compensation to obtain the non-key frames of the preliminary enhanced reconstruction; the deep reconstruction network module FDR-Net of the second stage carries out residual reconstruction by using frame difference information to recover details to obtain the finally reconstructed non-key frames. The method can significantly improve the reconstruction quality of the non-key frames, and high-quality video fast reconstruction is realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an efficient distributed video compressive sensing reconstruction system and method. BACKGROUND

[0002] Due to the fact that compressive sensing (CS) requires fewer measurements in signal sampling than the Nyquist sampling theory, it has been applied to image and video coding. The CS theory shows that if a signal is compressible or sparse in a certain transform domain, it can be accurately reconstructed from a small number of measurements. Mathematically, x e R N×1 represents an original signal, and Φ e R M×N is a linear random sampling matrix. Therefore, the CS measurement value y e R M×1 can be obtained by linear projection y = Φx, and the ratio of M to N, γ, is called the sampling rate. The inverse problem of CS is a typical ill-posed problem. How to design the sampling matrix and how to accurately reconstruct the original signal from a small number of measurements are still the main challenges faced by the CS theory.

[0003] Most existing video compressive sensing works are based on a distributed compressive video sensing architecture, in which frames are sampled individually, and recovered jointly by exploiting the spatio-temporal correlation. Multi-hypothesis performs residual reconstruction between the target frame and its multi-hypothesis predictions, and since the residual is more compressible than the original signal, multi-hypothesis achieves good performance in both recovery quality and computational complexity, and becomes a milestone in video compressive sensing, and has attracted extensive attention. As an alternative to the multi-hypothesis method, some techniques focus on sparsity and exploit the spatio-temporal correlation by constructing an adaptive dictionary. However, the above methods usually recover the original signal through iterative optimization, resulting in high computational complexity and unstable recovery results.

[0004] Due to the success of convolutional neural networks in image CS, some video compressive sensing methods based on convolutional neural networks have been proposed recently, and have achieved better performance than traditional methods. Some of these methods use long short-term memory networks to extract motion features between adjacent frames for video compressive sensing reconstruction. Some methods develop a two-stage network with explicit motion compensation to obtain a more suitable sparse representation in the inter-frame group domain in order to better exploit the non-local correlation between frames. Some other methods do not use optical flow estimation, but use deformable convolution for temporal alignment, and fuse inter-frame information through 3D convolution to recover the target frame. Although the performance of these methods is promising, the additional hyperparameters and computations limit their flexibility and practical applications. SUMMARY

[0005] The application aims at solving the problems in the prior art and providing an efficient distributed video compressed sensing reconstruction system and method.

[0006] To achieve the above object, the technical scheme of the application is as follows: an efficient distributed video compressed sensing reconstruction system, comprising an encoding module and a decoding module.

[0007] The encoding module comprises:

[0008] A video frame image grouping module extracts video sequence frame images and forms a video image frame group;

[0009] A compressed encoding module respectively compresses and samples key frames and non-key frames in the video image frame group, obtains corresponding compressed results and transmits them to the decoding module;

[0010] The decoding module comprises:

[0011] A key frame reconstruction module, after the key frame compressed result reaches the decoding module, independently reconstructs the key frame by ISTA-Net ++ algorithm;

[0012] A non-key frame preliminary reconstruction module, using the non-key frame compressed result, preliminarily reconstructs the non-key frame by ISTA-Net ++ algorithm;

[0013] A non-key frame deep reconstruction module, using the key frame obtained after independent reconstruction and the non-key frame obtained after preliminary reconstruction, completes deep reconstruction of the non-key frame through a deep reconstruction network to obtain the finally reconstructed non-key frame;

[0014] A video frame image recombination module recombines the key frame obtained after independent reconstruction and the non-key frame obtained after final reconstruction to form a decoded video sequence.

[0015] In an embodiment of the application, the video frame image grouping module extracts video sequence frame images from the input video sequence and forms a GOP frame group, the first frame image in the frame group is a key frame, and the remaining frame images are non-key frames; the frame images in each frame group are transmitted in the order of key frame and non-key frame.

[0016] In an embodiment of the application, the compressed encoding module obtains measurement values by using a convolution layer with a predetermined kernel size and step length to perform block-based frame-by-frame compressed sampling, and the sampling process is represented as formula (1):

[0017] y=A(X)=W Φ *X (1)

[0018] In the formula, X is an original signal; y is a compressed sensing measurement value; A(X) represents compressed sensing sampling of the original signal X; WΦ Let be the convolution kernel, where the subscript Φ is a fixed random Gaussian matrix; * denotes the convolution operation. To obtain the convolution kernel W... Φ The fixed random Gaussian matrix Φ∈R M×N Reshape into M convolutional kernels, each with a kernel size of...

[0019] In one embodiment of the present invention, the keyframe reconstruction module and the non-keyframe preliminary reconstruction module respectively adopt ISTA-Net. ++ The initialization module and network reconstruction module in the algorithm complete the independent reconstruction and preliminary reconstruction of key frames and non-key frames.

[0020] In one embodiment of the present invention, ISTA-Net ++ The initialization module in the algorithm combines deep learning convolution operations and a pixel shuffle layer to address the dimensionality mismatch between the original image and its compressed sensing measurements; it is defined as shown in equation (2):

[0021]

[0022] In the formula, Y represents the initialized measurement value; AT(Y) represents the compressed measurement value; AT(Y) indicates that the compressed measurement value is initialized to match the dimension between the original image and its compressed sensing measurement value. The kernel is Φ. T This indicates that a fixed random Gaussian matrix Φ is transposed; PixelShuffle represents a tensor reshaping algorithm, which can take an image with input (N, C×upscale_factor2, H, W) and output it as (N, C, H×upscale_factor, W×upscale_factor) after passing it through the PixelShuffle layer, where N is the number of samples in the image, C is the number of channels, H is the width of the image, W is the height of the image, and upscale_factor is the scaling factor. To obtain... Put Φ T ∈R N×M The tensor is reconstructed into N convolutional kernels, each with a kernel size of 1×1×M; the pixel shuffle layer reshapes the N×1×1 tensor back into a tensor.

[0023] In one embodiment of the present invention, ISTA-Net ++The network reconstruction module in the algorithm is to expand ISTA and alternate K times between the dynamic gradient descent module and the dynamic proximal mapping module; in order to process multiple ratios simultaneously, a dynamic expansion strategy is used to expand the soft threshold iterative algorithm, the dynamic expansion strategy takes the compressed sensing ratio as input, and uses a global condition module to transmit the information of the input ratio to each stage;

[0024] The dynamic gradient descent module takes one of the outputs of the condition module, the iteration step size ρ k of the Kth stage and the output of the K-1th stage initialization module

[0025]

[0026] In the formula, R k is the output of the Kth stage dynamic gradient descent module, A represents the process of compressed sensing sampling using the measurement matrix, A T represents the initialization process to match the dimensions between the original image and its compressed sensing measurement value.

[0027] The dynamic proximal mapping module introduces an additional noise level mapping as input, so that it has flexibility to multiple degradation levels; the dynamic proximal mapping module is composed of two residual blocks and and two convolutional layers and The two convolutional layers are respectively used for feature extraction and reconstruction, and long skip connection is performed, and the output of the Kth stage dynamic proximal mapping module is represented as formula (4):

[0028]

[0029] In order to more flexibly process multiple ratios of a single model, ISTA-Net ++ The condition module in the algorithm is designed to predict the condition information transmitted to each stage; the output of the condition module is where ρ k is the iteration step size of the Kth stage, σ k is the noise size of the Kth stage, three fully connected layers are used in the condition module, the first two activation functions are relu, and Softplus is used as the last activation function, and the condition information generated by the condition module is represented as formula (5):

[0030]

[0031] In the formula, H CM(γ) represents the whole processing procedure of the condition module, wherein γ is a compressed sensing sampling rate.

[0032] In an embodiment of the present application, the non-key frame depth reconstruction module adopts a depth reconstruction network FMCDR-Net to compensate each non-key frame by using the nearest two key frames; the depth reconstruction network FMCDR-Net aims to utilize the time redundancy existing in the key frames and the non-key frames to improve the quality of the non-key frame reconstruction in the video compressed sensing; the depth reconstruction network FMCDR-Net comprises an FMC-Net network module and an FDR-Net network module, the former performs preliminary reconstruction enhancement on the non-key frame, and the latter performs depth reconstruction on the preliminary reconstruction enhanced non-key frame.

[0033] In an embodiment of the present application, the FMC-Net network module is composed of a feature extraction component, a memory component and a reconstruction component, and learns from the design concept of STMC-Net to enhance the reconstruction of the non-key frame through the space-time correlation of the key frame and the non-key frame; the input of the FMC-Net network module is the reconstructed key frame from the current video image frame group and the preliminary reconstructed non-key frame from the current video image frame group, and the output is the preliminary reconstruction enhanced non-key frame.

[0034] In an embodiment of the present application, the FDR-Net network module is composed of two convolutional layers and a plurality of residual blocks, and aims to effectively learn and utilize the frame difference information; two reconstructed key frames, one preliminary reconstruction enhanced non-key frame and three frame differences are mapped through feature fusion to form a six-channel input feature map, the first convolutional layer is used to extract features from the input feature map, then a plurality of residual blocks are used, and long skip connection is performed to preserve the original information while learning the residual; finally, the second convolutional layer is used to generate the final reconstructed non-key frame.

[0035] The present application also provides an efficient distributed video compressed sensing reconstruction method, which comprises an encoding process and a decoding process; wherein,

[0036] The encoding process comprises:

[0037] Video sequence frame images are extracted from the input video sequence to form a video image frame group;

[0038] The key frames and the non-key frames in the video image frame group are compressed and sampled respectively, and the corresponding compression results are transmitted to the decoding module;

[0039] The decoding process comprises

[0040] ISTA-Net ++ algorithm is used to independently reconstruct the key frame compression result;

[0041] ISTA-Net++ The algorithm performs preliminary reconstruction on the compression result of the non-key frame;

[0042] By using the key frame obtained after independent reconstruction and the non-key frame obtained after preliminary reconstruction, the depth reconstruction of the non-key frame is completed through the depth reconstruction network, and the finally reconstructed non-key frame is obtained.

[0043] The key frame obtained after independent reconstruction and the non-key frame obtained after final reconstruction are recombined to form a decoded video sequence.

[0044] Compared with the prior art, the method of the present application has the following beneficial effects: the method of the present application first uses ISTA-Net ++ The present application introduces a distributed video compression sensing system to independently reconstruct and preliminarily reconstruct the key frame and the non-key frame, fully utilizes the intra-frame correlation, and through the designed two-stage depth reconstruction network module, fully utilizes the inter-frame correlation to perform depth reconstruction on the non-key frame. The first-stage depth reconstruction network module FMC-Net performs motion compensation to obtain a preliminarily enhanced reconstructed non-key frame; the second-stage depth reconstruction network module FDR-Net uses frame difference information to perform residual reconstruction to recover details to obtain a finally reconstructed non-key frame. The method of the present application can significantly improve the reconstruction quality of the non-key frame, and realizes high-quality video fast reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The present application is a high-efficiency distributed video compression sensing reconstruction network overall architecture schematic diagram.

[0046] Figure 2 The present application is a video compression sensing reconstruction network module implementation and application flowchart.

[0047] Figure 3 The present application is an ISTA-Net ++ network architecture schematic diagram.

[0048] Figure 4 The present application is a reconstruction effect visual comparison diagram of the coastguard sequence under different sampling rates. DETAILED DESCRIPTION

[0049] The technical solutions of the present application will be specifically described below with reference to the accompanying drawings.

[0050] As shown in the drawings, the present application provides a high-efficiency distributed video compression sensing reconstruction system, which comprises an encoding module and a decoding module. Figure 1

[0051] The encoding module comprises:

[0052] ​a video frame image grouping module, extracting video sequence frame images to form a video image frame group;

[0053] a compression encoding module, respectively compressing and sampling key frames and non-key frames in the video image frame group, obtaining corresponding compression results and transmitting them to a decoding module;

[0054] The decoding module comprises:

[0055] a key frame reconstruction module, after the key frame compression result reaches the decoding module, the key frame is independently reconstructed by ISTA-Net ++ algorithm;

[0056] a non-key frame preliminary reconstruction module, using the non-key frame compression result, the non-key frame is preliminarily reconstructed by ISTA-Net ++ algorithm;

[0057] a non-key frame deep reconstruction module, using the key frame obtained after independent reconstruction and the non-key frame obtained after preliminary reconstruction, the non-key frame is deep reconstructed by a deep reconstruction network to obtain a final reconstructed non-key frame;

[0058] a video frame image recombination module, the key frame obtained after independent reconstruction and the non-key frame obtained after final reconstruction are recombined to form a decoded video sequence.

[0059] The video frame image grouping module extracts video sequence frame images from the input video sequence and forms a GOP frame group, the first frame image in the frame group is a key frame, and the remaining several frame images are non-key frames; the frame images in each frame group will be transmitted in the order of key frames and non-key frames.

[0060] The compression encoding module, unlike most methods using a fixed random Gaussian matrix, obtains measurement values by using convolution layers with special kernel size and step length for block-based frame-by-frame compression sampling, which not only reduces the computational complexity and storage requirements, but also can sample key frames and non-key frames at different sampling rates. The sampling process is represented as formula (1):

[0061] y=A(X)=W Φ *X (1)

[0062] In the formula, X is the original signal; y is the compression sensing measurement value; A(X) represents the compression sensing sampling of the original signal X; W Φ

[0063] is a convolution kernel, where the subscript Φ is a fixed random Gaussian matrix; * is a convolution operation. In order to obtain the convolution kernel W Φ , the fixed random Gaussian matrix Φ ∈ R M×N is reshaped into M convolution kernels, and the kernel size of each convolution kernel is

[0064] The keyframe reconstruction module and the non-keyframe preliminary reconstruction module both use ISTA-Net. ++ The initialization module and network reconstruction module in the algorithm complete the independent reconstruction and preliminary reconstruction of key frames and non-key frames.

[0065] See Figure 3 As shown, ISTA-Net ++ The initialization module in the algorithm combines deep learning convolution operations and a pixel shuffle layer to address the dimensionality mismatch between the original image and its compressed sensing measurements; it is defined as shown in equation (2):

[0066]

[0067] In the formula, Y represents the initialized measurement value; AT(Y) represents the compressed measurement value; AT(Y) indicates that the compressed measurement value is initialized to match the dimension between the original image and its compressed sensing measurement value. The kernel is Φ. T This indicates that a fixed random Gaussian matrix Φ is transposed; PixelShuffle represents a tensor reshaping algorithm, which can take an image with input (N, C×upscale_factor2, H, W) and output it as (N, C, H×upscale_factor, W×upscale_factor) after passing it through the PixelShuffle layer, where N is the number of samples in the image, C is the number of channels, H is the width of the image, W is the height of the image, and upscale_factor is the scaling factor. To obtain... Put Φ T ∈R N×M The tensor is reconstructed into N convolutional kernels, each with a kernel size of 1×1×M; the pixel shuffle layer reshapes the N×1×1 tensor back into a tensor.

[0068] See Figure 3 As shown, ISTA-Net ++ The network reconstruction module in the algorithm expands the traditional ISTA and alternates between the dynamic gradient descent module and the dynamic proximal mapping module K times. In order to process multiple ratios simultaneously, a dynamic expansion strategy is used to expand the soft thresholding iterative algorithm. The dynamic expansion strategy takes the compressed sensing ratio as input and uses a global condition module to pass the information of the input ratio to each stage.

[0069] The dynamic gradient descent module takes one of the outputs of the conditional module as the iteration step size ρ for the Kth stage. kand the output of the K-1th stage initialization module The fixed random Gaussian matrix Φ and the compressed measurements Y are input into the dynamic gradient descent module, and the output of the Kth stage is defined as shown in equation (3):

[0070]

[0071] where R k is the output of the Kth stage dynamic gradient descent module, A represents the process of compressive sensing sampling using the measurement matrix, and A T represents the initialization process to match the dimensions between the original image and its compressed sensing measurements.

[0072] The dynamic proximal mapping module introduces an additional noise level mapping as input, which has flexibility for multiple degradation levels; the dynamic proximal mapping module is composed of two residual blocks and and two convolutional layers and The two convolutional layers are used for feature extraction and reconstruction, respectively, and long skip connections are performed. The output of the Kth stage dynamic proximal mapping module is represented as shown in equation (4):

[0073]

[0074] To more flexibly handle multiple ratios of a single model, ISTA-Net ++ algorithm, a condition module is designed to predict the condition information transmitted to each stage, and the condition information transmitted to the dynamic gradient descent module and the dynamic proximal mapping module has great potential in improving the generalization performance; let the output of the condition module be where ρ k is the iteration step size of the Kth stage, and σ k is the noise size of the Kth stage. Three fully connected layers are used in the condition module, the first two activation functions are relu, and Softplus is used as the last activation function. The condition information generated by the condition module is represented as shown in equation (5):

[0075]

[0076] where H CM (γ) represents the entire processing process of the condition module, where γ is the compressive sensing sampling rate.

[0077] The non-key frame depth reconstruction module, due to the higher sampling rate, the key frame is reconstructed by ISTA-Net ++The reconstructed key frames have better reconstruction quality, so they can be introduced into the non-key frame recovery; the non-key frame depth reconstruction module uses the depth reconstruction network FMCDR-Net to compensate each non-key frame by using the nearest two key frames (from the current GOP and the next GOP);

[0078] The depth reconstruction network FMCDR-Net aims to utilize the temporal redundancy existing in key frames and non-key frames to improve the quality of non-key frame reconstruction in video compressive sensing; this method is based on the observation that the difference between frames contains valuable information that can be used to improve the quality of non-key frame reconstruction. The depth reconstruction network FMCDR-Net includes an FMC-Net network module and an FDR-Net network module, the former performs preliminary reconstruction enhancement on non-key frames, and the latter performs depth reconstruction on the preliminary reconstruction enhanced non-key frames.

[0079] The FMC-Net network module draws on the design concept of STMC-Net, thereby enhancing the reconstruction of non-key frames through the spatio-temporal correlation of key frames and non-key frames. This network module mainly consists of 3 components: a feature extraction component, a memory component, and a reconstruction component. Each component plays a crucial role in extracting, processing, and reconstructing the features of key frames and non-key frames. The feature extraction component consists of two convolutional layers and one max-pooling layer, which extracts spatial features from key frames and non-key frames, respectively. The memory component consists of two convolutional layers, which perform feature connection on the spatial features from key frames and non-key frames to form rich feature representations and capture spatial and temporal correlations. The reconstruction component consists of two convolutional layers, the first convolutional layer is used to process the memory-enhanced features, and the second convolutional layer is used to output the preliminary enhanced non-key frames, and finally a Sigmoid function is used for normalization operation. FMC-Net utilizes the memory mechanism to capture rich contextual information and better utilize the spatio-temporal correlation between key frames and non-key frames, achieving significant results in the preliminary enhancement reconstruction of non-key frames.

[0080] The FDR-Net network module is composed of two convolutional layers and multiple residual blocks, and is designed to effectively learn and utilize frame difference information; the network adopts six input channels: two reconstructed key frames, one preliminary enhanced reconstructed non-key frame, and three frame difference maps. This multi-channel input allows the network to capture spatial and temporal relationships between frames. We compare the reconstructed key frame from the current GOP, the reconstructed key frame of the next GOP, and the preliminary enhanced reconstructed non-key frame from the current GOP to obtain three frame difference maps. Then, the two reconstructed key frames, the preliminary enhanced reconstructed non-key frame, and the three frame difference maps are fused to form a six-channel input feature map. The first convolutional layer is used to extract features from the input feature map, and then multiple residual blocks are used, with long skip connections, to preserve the original information while learning the residual. Finally, the second convolutional layer is used to generate the final reconstructed non-key frame.

[0081] The key of FDR-Net lies in its ability to effectively utilize frame difference information. By inputting key frames and their differences with the target non-key frame into the network, the model can more accurately predict the non-key frame. The residual blocks in the network facilitate the learning of complex mappings by focusing on residual information, simplifying the learning process and enhancing convergence. The use of frame difference maps in the network is particularly useful because by integrating multiple information sources (key frames and their differences), it captures temporal redundancy between frames, provides additional information that spatial-only data cannot provide, and can fully utilize the correlation between frames. Second, residual connections help to learn the subtle changes between frames, which is crucial for high-quality reconstruction of non-key frames.

[0082] The application also provides an efficient distributed video compression sensing reconstruction method, including an encoding process and a decoding process; wherein,

[0083] The encoding process includes:

[0084] Extracting video sequence frame images from the input video sequence to form a video image frame group;

[0085] Compressing and sampling the key frames and non-key frames in the video image frame group respectively, obtaining the corresponding compression results and transmitting them to the decoding module;

[0086] The decoding process includes

[0087] Using ISTA-Net ++ algorithm to independently reconstruct the key frame compression results;

[0088] Using ISTA-Net ++ algorithm to preliminarily reconstruct the non-key frame compression results;

[0089] Using the keyframes obtained after independent reconstruction and the non-keyframes obtained after preliminary reconstruction, a deep reconstruction network is used to complete the deep reconstruction of the non-keyframes, resulting in the final reconstructed non-keyframes.

[0090] The decoded video sequence is formed by recombining the keyframes obtained from independent reconstruction and the non-keyframes obtained from final reconstruction.

[0091] like Figure 2 As shown, the implementation and application process of the system and method of this invention includes:

[0092] Step 1: Obtain the network training and test datasets using UCF-101, which contains 13,320 video clips at a resolution of 320×240. This dataset is randomly split into 80% for training and 20% for validation. Due to GPU memory limitations, all frames are cropped into 160×160 patches, with only the luminance components retained.

[0093] Step 2: Construct the network model for video compression and perception reconstruction of the system of this invention. The overall network structure is as follows: Figure 1 As shown, it consists of three parts: a compressed sampling network, a preliminary reconstruction network, and a two-stage deep reconstruction network.

[0094] Step 3: Set the network model loss function, hyperparameters, and optimization method. In each training phase of the network, the mean squared error is used as the loss function.

[0095]

[0096] Where B represents the batch size. For the corresponding preliminary reconstruction of non-key frames or preliminary enhancement and reconstruction of non-key frames, k i Here, θ represents the corresponding reconstructed keyframe, Θ represents the parameters of the corresponding network, and represents the network output. This paper uses the ADAM optimizer to update the network parameters, setting the learning rate to 0.0001 and the batch size to 12, and applying the ReLU activation function to the output of each convolutional layer.

[0097] Step 4: Train the network model using the prepared training set. To speed up convergence, we first pre-train each sub-network individually and sequentially, and then jointly train all of these networks.

[0098] Step 5: Use the trained network model to perform compressed sensing reconstruction of the video, and use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as evaluation metrics to verify the network performance: Calculate the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) between the original image and the reconstructed image.

[0099] As an implementable manner, the model of the application takes the data set UCF-101 as a training set, and the standard CIF video development sequence is used to test the model.

[0100] As an implementable manner, the objective evaluation index includes the full reference evaluation index peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The higher the PSNR value, the closer the output image and the reference image in content. The higher the SSIM value, the more similar the two images in structure.

[0101] The embodiment verifies and analyzes the method, more specifically: in the experiment, the method is compared with nine representative video reconstruction methods in terms of reconstruction quality and reconstruction speed, including three traditional methods: MH-BCS-SPL, RRS and GSR-InterF-SSIM, and six deep learning-based methods: CSVideoNet, PRCVSNet, 2sER-VGSR-Net, STM-Net, ImrNet and DUMHAN.

[0102] All experiments are implemented on pytorch, and the experimental platform is configured as Intel Core i5-12400F CPU and RTX 4060 GPU.

[0103] The sequence uses the standard QCIF video development sequence football, hall, mother-daughter, soccer, tennis and ice sequence.

[0104] In Table 1, we can see that the reconstruction quality of FMCDR-Net is better than that of other methods. When the key frame sampling rate is 0.7 and the non-key frame sampling rate is 0.1, compared with the other three methods, the average PSNR is increased by 4.63, 4.24 and 2.84, and the average SSIM is increased by 0.09, 0.08 and 0.05.

[0105] When the key frame sampling rate is 0.7 and the non-key frame sampling rate is 0.05, compared with the other three methods, the average PSNR is increased by 4.64, 7.51 and 3.21, and the average SSIM is increased by 0.10, 0.15 and 0.07.

[0106] When the key frame sampling rate is 0.7 and the non-key frame sampling rate is 0.01, compared with the other three methods, the average PSNR is increased by 6.22, 9.29 and 4.54, and the average SSIM is increased by 0.18, 0.29 and 0.11.

[0107] From the above result analysis, in the case of the same sampling rate of key frames, the lower the sampling rate of non-key frames, the more significant the advantage of the video compressive sensing reconstruction method of the application in reconstruction quality compared with the other three methods.

[0108] In Table 1, we can get that the reconstruction speed of FMCDR-Net is much lower than that of other methods. The average time for restoring the video frames of the above six video sequences is only 9 ms, realizing millisecond-level reconstruction. While the reconstruction time of video frames of other methods is higher than several tens of seconds at any sampling rate, and the reconstruction delay thereof cannot meet the real-time requirement. Compared with other existing video compressive sensing reconstruction methods, the video compressive sensing reconstruction method of the application breaks the bottleneck of slow recovery speed of traditional compressive sensing.

[0109] Table 1 Comparison of PSNR, SSIM and reconstruction speed of video frames of six QCIF video sequences reconstructed by the traditional method

[0110]

[0111] We will compare the designed FMCDR-Net with the six recently developed methods using deep learning: CSVideoNet, PRCVSNet, 2sER-VGSR-Net, STM-Net, ImrNet and DUMHAN. In order to make a fair comparison, FMCDR-Net is trained and evaluated on the same dataset as the aforementioned methods.

[0112] In Table 2, we can get the comparison results on the UCF-101 dataset, and FMCDR-Net achieves better average PSNR and SSIM values at each sampling rate.

[0113] When the sampling rate of key frames is 0.2 and the sampling rate of non-key frames is 0.037, compared with CSVideoNet, PRCVSNet, 2sER-VGSR-Net and STM-Net, the average PSNR is increased by 5.81, 1.59, 1.45 and 0.18 respectively, and the average SSIM is increased by 0.12, 0, 0.04 and 0.00 respectively.

[0114] When the sampling rate of key frames is 0.2 and the sampling rate of non-key frames is 0.018, compared with CSVideoNet, PRCVSNet, 2sER-VGSR-Net and STM-Net, the average PSNR is increased by 6.69, 2.85, 2.18 and 0.64 respectively, and the average SSIM is increased by 0.14, 0, 0.06 and 0.00 respectively.

[0115] When the key frame sampling rate is 0.2 and the non-key frame sampling rate is 0.009, compared with CSVideoNet, PRCVSNet, 2sER-VGSR-Net and STM-Net, the average PSNR is increased by 6.74, 4.11, 2.58 and 0.99 respectively, and the average SSIM is increased by 0.16, 0, 0.08 and 0.01 respectively.

[0116] Although FMCDR-Net is slightly inferior to ImrNet and DUMHAN in terms of PSNR, the two methods improve the PSNR by increasing the depth of the model. This will inevitably lead to the increase of the skeleton of the model, and increase the difficulty of model training and practical application. As can be seen from Table 2, the memory consumption of FMCDR-Net is lower than that of the models of ImrNet and DUMHAN, and is even only one third of that of DUMHAN.

[0117] In terms of running speed, the results of FMCDR-Net are not inferior due to the adoption of the simplified architecture. All experimental results confirm the effectiveness of our method, even in the case of low sampling rate and large time interval.

[0118] Table 2 Comparison of PSNR, SSIM and reconstruction speed of deep learning-based methods for reconstructing video frames on the UCF-101 dataset

[0119]

[0120] Figure 4 Figure 6 is a visual comparison chart of the reconstruction effect of the method of the present application on the coastguard sequence under different sampling rates.

[0121] The above embodiments of the present application are merely examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments are not required to be exhausted. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. An efficient distributed video compressive sensing reconstruction system, characterized in that, The encoding module and the decoding module are included. The encoding module includes: The video frame image grouping module extracts video sequence frame images from the input video sequence and forms GOP frame groups, the first frame image in the frame group is a key frame, and the remaining several frame images are non-key frames; the frame images in each frame group will be transmitted in the order of key frames and non-key frames. The compression encoding module obtains measurement values by using a convolution layer with a predetermined kernel size and step for block-based frame-by-frame compression sampling, and the sampling process is represented as formula (1): The FMC-Net network module is composed of a feature extraction component, a memory component, and a reconstruction component, and enhances the reconstruction of non-key frames through the spatiotemporal correlation of key frames and non-key frames; the input of the FMC-Net network module is the reconstructed key frame from the current video image frame group and the preliminary reconstructed non-key frame from the current video image frame group, and the output is the preliminary reconstruction enhanced non-key frame. The key frame reconstruction module, after the key frame compression result reaches the decoding module, independently reconstructs by ISTA-Net ++ algorithm a non-key frame preliminary reconstruction module, using the non-key frame compression result, performing preliminary reconstruction by ISTA-Net ++ algorithm; The FDR-Net network module is composed of two convolution layers and multiple residual blocks; two reconstructed key frames, one preliminary reconstruction enhanced non-key frame, and three frame differences are mapped through feature fusion to form a six-channel input feature map, the first convolution layer is used to extract features from the input feature map, then multiple residual blocks are used, and long skip connections are used to preserve the original information while learning the residual; finally, the second convolution layer is used to generate the final reconstructed non-key frame. The encoding module and the decoding module are included. The encoding process includes:

2. The efficient distributed video compressive sensing reconstruction system according to claim 1, wherein, Extract video sequence frame images from the input video sequence and form video image frame groups; 3. The efficient distributed video compressive sensing reconstruction system of claim 1, wherein, Respectively compress and sample the key frames and non-key frames in the video image frame groups, and transmit the corresponding compression results to the decoding module; y = A(X) = W Φ *X (1) where X is the original signal; y is the compressed sensing measurement value; A(X) represents the compressed sensing sampling of the original signal X; W Φ is the convolution kernel, where the subscript Φ is a fixed random Gaussian matrix; * is the convolution operation; in order to obtain the convolution kernel W Φ , a fixed random Gaussian matrix Φ ∈ R M×N is reshaped into M convolution kernels, and the kernel size of each convolution kernel is 4. The efficient distributed video compressive sensing reconstruction system of claim 1, wherein, The key frame reconstruction module and the non-key frame preliminary reconstruction module respectively adopt ISTA-Net ++ The initialization module and the network reconstruction module in the algorithm complete independent reconstruction and preliminary reconstruction of the key frame and the non-key frame.

5. The efficient distributed video compressive sensing reconstruction system of claim 4, wherein, ISTA-Net ++ The initialization module in the algorithm combines the convolution operation of deep learning and the pixelshuffle layer to process the dimension mismatch problem between the original image and its compressed sensing measurement value; defined as shown in formula (2): wherein, is the initialized measurement value; Y is the compressed measurement value; A T (Y) represents an initialization process on the compressed measurement value, matching the dimensionality between the original image and its compressed sensing measurement value; is the convolution kernel, subscript Φ T represents transposing the fixed random Gaussian matrix Φ; PixelShuffle represents a tensor reshaping algorithm, which outputs an image of (N, C x upscale_factor2, H, W) as (N, C, H x upscale_factor, W x upscale_factor) after the PixelShuffle layer, where N is the sample number of the image, C is the channel number, H is the width of the image, W is the height of the image, and upscale_factor is the scale factor; in order to obtain Φ T ∈R N×M is reconstructed into N convolution kernels, and the kernel size of each convolution kernel is 1 x 1 x M; wherein the PixelShuffle layer reshapes the tensor N x 1 x 1 into the tensor 6. The efficient distributed video compressive sensing reconstruction system of claim 4, wherein, ISTA-Net ++ The network reconstruction module in the algorithm is to unfold ISTA and alternate K times between the dynamic gradient descent module and the dynamic proximal mapping module; in order to process multiple ratios simultaneously, a dynamic unfolding strategy is used to unfold the soft threshold iteration algorithm, the dynamic unfolding strategy takes the compressed sensing ratio as input, and uses a global condition module to pass the information of the input ratio to each stage; The dynamic gradient descent module defines the iteration step size p of the Kth stage of the conditional module as one of the outputs of the conditional module k and the output of the K-1th stage initialization module The output of the dynamic gradient descent module at the Kth stage is defined as shown in equation (3) with the fixed random Gaussian matrix Φ and the compressed measurements Y as inputs In the formula, R k is the output of the dynamic gradient descent module of the Kth stage, A represents the process of using a measurement matrix for compressed sensing sampling, A T represents an initialization process to match the dimensions between the original image and its compressed sensing measurement values; The dynamic near-end mapping module introduces additional noise level mapping As input, it has flexibility to multiple degradation levels; The dynamic near-end mapping module consists of two residual blocks and And two convolutional layers and Two convolutional layers are used for feature extraction and reconstruction respectively, and long skip connection is performed, and the output of the dynamic near-end mapping module of the Kth stage is represented as formula (4): To handle multiple ratios of a single model more flexibly, ISTA-Net ++ A condition module is designed in the algorithm to predict the condition information transmitted to each stage; Output of the conditioning module where p k is the iteration step size of the Kth stage, s k is the noise size of the Kth stage, three fully connected layers are used in the conditioning module, the first two activation functions are relu, and Softplus is used as the last activation function. The conditional information generated by the conditioning module is represented as shown in equation (5): In the formula, H CM (γ) represents the entire processing procedure of the conditional module, where γ is the compression sensing sampling rate.

7. The efficient distributed video compressive sensing reconstruction system of claim 1, wherein, The decoding process includes 8. The efficient distributed video compressive sensing reconstruction system of claim 1, wherein, ​ 9. An efficient distributed video compressive sensing reconstruction method, characterized in that, ​ ​ ​ ​ ​ using ISTA-Net ++ an algorithm to independently reconstruct the key frame compression results; using ISTA-Net ++ The algorithm performs preliminary reconstruction on the non-key frame compression result; The key frames obtained after independent reconstruction and the non-key frames obtained after preliminary reconstruction are used to complete depth reconstruction of the non-key frames through a depth reconstruction network to obtain finally reconstructed non-key frames; The key frames obtained after independent reconstruction and the non-key frames obtained after final reconstruction are recombined to form a decoded video sequence; The implementation of completing depth reconstruction of the non-key frames through the depth reconstruction network to obtain finally reconstructed non-key frames is that a depth reconstruction network FMCDR-Net is used to compensate each non-key frame through the two nearest key frames; the depth reconstruction network FMCDR-Net aims to use the time redundancy existing in the key frames and the non-key frames to improve the quality of reconstruction of the non-key frames in video compression sensing; the depth reconstruction network FMCDR-Net comprises an FMC-Net network module and an FDR-Net network module, the former is used to preliminarily reconstruct and enhance the non-key frames, and the latter is used to perform depth reconstruction on the non-key frames preliminarily reconstructed and enhanced.

Images