Video compressed sensing reconstruction method and model based on lightweight 3D deformable convolution

By adopting a lightweight 3D deformable convolutional network in video compression perception, high-quality reconstruction is carried out using inter-frame correlation, which solves the problem of insufficient utilization of inter-frame correlation in the prior art, and realizes efficient video compression perception reconstruction. And reduce the parameters and calculation amount.

CN118984397APending Publication Date: 2024-11-19DALIAN UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411020098.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing video compression perception methods are insufficient to utilize inter-frame correlation, resulting in a large gap in reconstruction quality between frames, and the excessive amount of parameters introduced by 3D deformable convolution is not conducive to model training.

Method used

Adopting a lightweight 3D deformable convolutional network, by introducing a combination of multi-layer residual connections and specific convolution kernel sizes, reducing the number of parameters and calculations, improving model computing efficiency, and achieving high-quality video compression perception at low sampling rates reconstruction.

Benefits of technology

It significantly improves the utilization of correlation between video frames, reduces the quality gap between frames, improves the quality of video compression-aware reconstruction and the computing efficiency of the model, and is scalable and can be integrated into other reconstruction algorithms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984397A_ABST
    Figure CN118984397A_ABST
Patent Text Reader

Abstract

According to the video compressed sensing reconstruction method and model based on the lightweight 3D deformable convolution provided by the invention, the lightweight 3D deformable convolution is introduced into a network, and high-quality video compressed sensing reconstruction can be realized at a low sampling rate. The model structure comprises a coding part which is composed of a trainable observation matrix and is used for carrying out compressed sampling on an input video frame; the decoding part is composed of a trainable reconstruction matrix and is used for preliminarily recovering the compressed and sampled video frame; the CNN block module is composed of a plurality of convolution layers, a pooling layer and a deconvolution layer; the 3D block module is used for enabling all video frames in a further reconstruction result output by the CNN block module to adaptively utilize key frame information to carry out overall reconstruction; comprising a lightweight 3D deformable convolutional network with two layers of 32 convolution kernels and a lightweight 3D deformable convolutional network with one layer of 1 convolution kernel, and the lightweight 3D deformable convolutional networks with the two layers of 32 convolution kernels are connected through multiple layers of residual errors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of video compression technology, and mainly relates to a video compression perception reconstruction method and model based on lightweight 3D deformable convolution, which can be used to achieve video compression perception reconstruction with better visual effect. Background Art

[0002] Video compression sensing is a technology that can compress scene video while sampling it. The sampling end with limited performance sends the compressed video signal to the reconstruction end with relatively rich resources. This can reduce data storage space and transmission bandwidth, and greatly reduce the cost of the sampling end.

[0003] Existing compressed sensing methods are divided into compressed sensing methods based on depth priors and compressed sensing methods based on deep learning. The method based on deep learning can achieve rapid reconstruction of videos, but requires a large amount of training data, while the method based on depth prior information requires manual design of some parameters. Both methods make insufficient use of the inter-frame correlation of videos, and the quality of the reconstructed video frames varies greatly.

[0004] 3D deformable convolution can make full use of the spatiotemporal information in videos and has great application prospects in video reconstruction tasks. Video super-resolution reconstruction based on 3D deformable convolution has achieved good visual effects. However, the offset introduced by 3D deformable convolution leads to a huge amount of parameters, which is not conducive to model training. Summary of the invention

[0005] In order to overcome the problem of large reconstruction quality gap between frames caused by insufficient utilization of inter-frame correlation in existing video compressed sensing reconstruction, the present application provides a video compressed sensing reconstruction method and model based on lightweight 3D deformable convolution, which introduces lightweight 3D deformable convolution into the network and can achieve high-quality video compressed sensing reconstruction at low sampling rate.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A video compression sensing reconstruction model based on lightweight 3D deformable convolution, the model structure includes:

[0008] Coding part: As the observation part, it is composed of a trainable observation matrix, which is used to compress and sample the input video frames. The observation matrix includes sampling matrices of non-key frames and key frames.

[0009] Decoding part: as the reconstruction part, it is composed of a trainable reconstruction matrix, which is used to initially restore the video frames after compression sampling. The reconstruction matrix includes the reconstruction matrices of non-key frames and key frames.

[0010] CNN block module: It consists of multiple convolutional layers, a pooling layer, and a deconvolution layer, and is used to further reconstruct the video frames of the decoded part output;

[0011] 3D block module: used to adaptively reconstruct all video frames in the further reconstruction results output by the CNN block module using key frame information; it includes two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels and one layer of lightweight 3D deformable convolutional network with 1 convolutional kernel. The two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels are connected by multiple layers of residual connections.

[0012] Furthermore, the lightweight 3D deformable convolutional network is implemented by combining a 3D deformable convolution with a convolution kernel size of "3×3×1" in a spatial dimension and a 3D convolution with a convolution kernel size of "1×1×3" in a temporal dimension.

[0013] Furthermore, the convolution kernel sizes of multiple convolution layers of the CNN block module are all 3×3, the strides are all 1×1, the convolutions are all padded with zeros, and ELU is used for activation; a pooling layer of the CNN block module uses maximum pooling; a deconvolution layer of the CNNblock module has a convolution kernel size of 2×2, a stride of 2×2, and no zero padding for the convolution.

[0014] The present application also provides a video compression sensing reconstruction method based on lightweight 3D deformable convolution, which is implemented based on the above model and includes the following steps:

[0015] S1: Prepare the dataset, split the video into video frames and adjust the size;

[0016] S2: The video frames are divided into key frames and non-key frames, and they are sampled and reconstructed with different sampling rates and reconstruction matrices respectively;

[0017] S3: The reconstruction result of each frame of video is further reconstructed through the CNN block module;

[0018] S4: Combine the further reconstruction results of all frames and input them into the lightweight 3D deformable convolutional network with two layers of 32 convolution kernels of the 3D block module for overall reconstruction and output the reconstructed feature map;

[0019] S5: The overall reconstructed feature map is passed through a lightweight 3D deformable convolutional network with one convolution kernel in the 3D block module to obtain the final video reconstruction result.

[0020] Furthermore, step S3 is specifically as follows: the video frame first passes through multiple convolution layers, and the number of feature map channels changes from 1 to 32; then passes through the pooling layer, and the feature map size becomes smaller; then passes through multiple convolution layers, and the number of channels changes from 32 to 64; then passes through the deconvolution layer, the feature map size is restored, and the number of channels is still 64; then passes through multiple convolution layers, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained through the previous convolution layers in the channel dimension through a cross connection to obtain a 64-channel feature map; this 64-channel feature map is then passed through multiple convolution layers, and the number of feature map channels changes from 64 to 1.

[0021] Furthermore, steps S4 and S5 are specifically as follows: the further reconstruction results of all frames obtained in step S3 are used as five-dimensional 3D feature maps and input into the 3D block module. The five-dimensional 3D feature map of 1 channel is first passed through a lightweight 3D deformable convolutional network with 32 convolution kernels to obtain a 32-channel output feature map. The 32-channel output feature map is passed through a lightweight 3D deformable convolutional network with 32 convolution kernels of five layers of residual connections to obtain a 32-channel feature map. The 32-channel feature map is spliced ​​with the 32-channel feature map obtained by the lightweight 3D deformable convolutional network with 32 convolution kernels for the first time in the channel dimension through a crossing connection to obtain a 64-channel 3D feature map. The 3D feature map is passed through a layer of lightweight 3D deformable convolutional network with 1 convolution kernel to obtain an output feature map of one channel, which is the reconstruction result of all frames.

[0022] Furthermore, the method further includes: extracting the fourth frame from the reconstruction results of all frames, and calculating the mean square error between the reconstructed fourth frame and the fourth frame of the original video as the loss function.

[0023] The present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned video compression sensing reconstruction method is implemented.

[0024] The present application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned video compression sensing reconstruction method is implemented.

[0025] The present application also provides a video processing system, including a video input module, a model as described above, and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs the video.

[0026] The beneficial effects of the present invention include:

[0027] By introducing lightweight 3D deformable convolution, the present invention can more effectively utilize the correlation between video frames, so that all video frames can adaptively use key frame information for overall reconstruction, especially at low sampling rates, significantly improving the quality of video compression sensing reconstruction and reducing the gap in reconstruction quality between frames. Compared with traditional 3D convolutional networks, the use of lightweight 3D deformable convolutional networks reduces the number of parameters and the amount of calculation, and improves the operational efficiency of the model.

[0028] In summary, the present invention introduces lightweight 3D deformable convolution after each frame of video is further reconstructed to help non-key frames with slightly poorer quality to adaptively use the information of key frames in a more efficient way to improve the reconstruction accuracy, so that the reconstruction effect of non-key frames of the video is improved, and the overall reconstruction effect of the video is improved. In addition, the lightweight 3D deformable convolution proposed by the present invention can be integrated into other reconstruction algorithms to improve the accuracy of other reconstruction algorithms. Therefore, this method is scalable and can provide an optional solution for other video reconstruction tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the implementation process of lightweight 3D deformable convolution;

[0030] Figure 2 It is the overall structural diagram of the network model;

[0031] Figure 3 yes Figure 2 CNN block module structure diagram in;

[0032] Figure 4 yes Figure 2 3D block module structure diagram in;

[0033] Figure 5 This is a comparison of the reconstructed visual effects of different methods on the 10th frame of the dataset Flower when the sampling rate is 0.5 / 0.1;

[0034] Figure 6 It is a comparison chart of the reconstructed visual effects at different sampling rates on 6 video sequences. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0037] This application provides a video compression sensing reconstruction method and model based on lightweight 3D deformable convolution. Lightweight 3D deformable convolution is introduced into the network to achieve high-quality video compression sensing reconstruction at a low sampling rate, overcoming the problem of large reconstruction quality gap between frames caused by insufficient utilization of inter-frame correlation in existing video compression sensing reconstruction. The overall implementation process of this application is:

[0038] (1) Preparation of the dataset: The videos in the dataset are segmented into individual video frames through the video segmentation function. The video frames are resized to 160×160 and grouped into 8 frames into a folder.

[0039] (2) After a set of video frames are input into the network model, they are divided into key frames and non-key frames. The key frames and non-key frames are sampled and reconstructed with different sampling matrices and reconstruction matrices. Taking the key frame with a sampling rate of 0.5 as an example, the key frame of size 160×160 is divided into non-overlapping blocks of size 32×32 and transformed into a two-dimensional vector (25, 1024). This two-dimensional vector is multiplied with the trainable sampling matrix P (1024, 1024×0.5) to obtain the sampling result (25, 512). The sampling result is multiplied with the trainable reconstruction matrix R (512, 1024) to obtain (25, 1024), and then the initial reconstruction result (160, 160) is obtained after deformation.

[0040] (3) The initial reconstruction result of each frame is reconstructed through a CNN block. Specifically, the CNN block consists of multiple convolutional layers, a pooling layer and a deconvolution layer. Each frame will be further reconstructed through the CNN block.

[0041] (4) The further reconstruction results of all frames are combined together and input into the 3D block as a five-dimensional video signal. Through a two-layer lightweight 3D deformable convolutional network with five layers of residual connections, all frames will make full use of the rich features of the key frames for further reconstruction.

[0042] (5) The feature map output by the two-layer lightweight 3D deformable convolutional network with five layers of residual connections is passed through a layer of lightweight 3D deformable convolutional network to obtain a 1-channel output result.

[0043] Example 1

[0044] This embodiment provides a video compression sensing reconstruction model based on lightweight 3D deformable convolution, including an encoding part and a decoding part. The encoding part is an observation part, and the decoding part is a reconstruction part.

[0045] The encoding part consists of a trainable observation matrix, which is used to compress and sample the input video frames. Its size is (B 2 , B 2 × sampling rate), where B is the block size, and the observation matrix includes P and P_key, where P and P_key are the sampling matrices of non-key frames and key frames respectively;

[0046] The decoding part consists of a trainable reconstruction matrix, which is used to initially restore the compressed sampled video frame. Its size is (B 2 × sampling rate, B 2 ), where B is the block size, the reconstruction matrix includes R and R_key, R and R_key are the reconstruction matrices of non-key frames and key frames respectively;

[0047] The reconstruction results of the key frames and non-key frames after the reconstruction matrix are further reconstructed through a CNN block. The module structure of the CNN block is as follows: Figure 3 As shown in the figure: it consists of multiple convolutional layers, a pooling layer, and a deconvolution layer, which is used to further reconstruct the decoded part of the output video frames; the convolution kernel size of the multiple convolutional layers of the CNN block module is 3×3, the step size is 1×1, the convolution is padded with zeros, and ElU is used for activation; a pooling layer of the CNN block module uses maximum pooling; a deconvolution layer of the CNN block module has a convolution kernel size of 2×2, a step size of 2×2, and no zero padding for the convolution.

[0048] The 3D block module is used to adaptively reconstruct all video frames in the further reconstruction results output by the CNN block module using key frame information; it includes two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels and one layer of lightweight 3D deformable convolutional network with 1 convolutional kernel. The two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels are connected by multiple layers of residual connections.

[0049] The lightweight 3D deformable convolution network is implemented by combining a 3D deformable convolution with a convolution kernel size of "3×3×1" in the spatial dimension and a 3D convolution with a convolution kernel size of "1×1×3" in the temporal dimension. The same effect as the 3D deformable convolution with a convolution kernel size of "3×3×3" is achieved with a lower number of parameters and calculations.

[0050] Example 2

[0051] This embodiment provides a video compression sensing reconstruction method based on lightweight 3D deformable convolution. Figure 1 , the implementation steps are as follows:

[0052] Step 1: Prepare the data set.

[0053] 1a) Install Python on your computer and install the third-party libraries cv2, os and shutil under Python;

[0054] 1b) Download the 3D deformable module dcn and run the bash make.sh command to generate the D3D module; and modify dcn / modules / deformable_conv.py to add the code for dimension=HWT and assign the offset in this case.

[0055] 1c) Download the UCF-101 dataset, which contains 101 human action videos with a video size of 320×240. Select the first 30 videos, the first 25 as the training set, and the last 5 as the validation set. Use cv2.videoCapture() to open the video, use cv2.ressize() to resize the video to 160×160, and then use os.makedirs() to create a folder and put every 8 frames of video in a subfolder. Name them frame_1, frame_2, etc. in the chronological order of the video frames. All subfolders constitute the training dataset and the validation dataset.

[0056] 1d) Cut the same position of each video sequence of the six standard CIF datasets "Akiyo, Coastguard, Mother_daughter, Paris, Flower, Bus" into 160×160 blocks. Since the size of the CIF dataset is 352×288, it is cut from the upper left to the upper right, and then from the lower left to the lower right, with a step length of 96, and cut into 6 blocks. Therefore, an original video of 8 frames can be cut into 6 blocks of 8 frames with a size of 160×160. Multiple subfolders after cutting are used as test sets. The overall test results of an original video of 8 frames can be obtained by splicing the test results of the blocks.

[0057] Step 2: Reconstruct the model through video compression sensing.

[0058] Taking the key frame as an example, it first passes through multiple convolutional layers, the number of convolution kernels are 1, 1, 32, 32, 32, and the number of feature map channels changes from 1 to 32; after the pooling layer, the feature map size becomes smaller; after multiple convolutional layers, the number of convolution kernels are 32, 32, 64, 64, 64, and the number of channels changes from 32 to 64; after the deconvolution layer, the number of convolution kernels is 64, the feature map size is restored, and the number of channels is still 64; after multiple convolutional layers, the number of convolution kernels is 64, 64, 32, 32, 32, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained by the first five convolutional layers through cross-connections in the channel dimension to obtain a 64-channel feature map; this 64-channel feature map passes through multiple convolutional layers, the number of convolution kernels is 64, 64, 32, 32, 32, 1, and the number of feature map channels changes from 64 to 1. After the above steps, the further reconstruction result of each frame is obtained. The further reconstruction results of all frames are input into the 3D block as five-dimensional feature maps. First, the five-dimensional 3D feature map of one channel is passed through 32 lightweight 3D deformable convolution kernels to obtain a 32-channel output feature map. The output feature map of the 32 channels is passed through a five-layer residual connected lightweight 3D deformable convolution network to obtain a 32-channel feature map. The 32-channel feature map and the 32-channel feature map obtained by the first lightweight 3D deformable convolution are spliced ​​in the channel dimension through a cross connection to obtain a 64-channel 3D feature map; this feature map is passed through three lightweight 3D deformable convolution layers with the number of convolution kernels being 32, 32, and 1 respectively to obtain a one-channel output feature map, which is the reconstruction result of all frames.

[0059] The fourth frame is extracted from the reconstruction results of all frames, and the mean square error between the reconstructed fourth frame and the fourth frame of the original video is calculated as the loss function.

[0060]

[0061] MSE is the mean square error between the reconstructed frame and the original frame, y is the original video frame, and y' is the reconstructed video frame.

[0062] Step 3, write the network project folder.

[0063] The network project folder includes: 3D deformable convolution module, network structure, network training and model testing files, dataset loading and preprocessing files, and hyperparameter settings. The steps are as follows:

[0064] 3a) Create a network project folder and add the 3D deformable module dcn

[0065] 3b) Design the network structure according to step 2, create a network structure file model.py in the project folder, and write the network structure in this file;

[0066] 3c) Create a network training file train_DDP.py in the network project folder, write the network training process in this file, and add hyperparameter information, batchsize = 1, and the learning rate lr is initially set to 1×10 -4 Every 10 epochs, the learning rate decays to 0.8 times of the original value. A total of 100 epochs are trained. The sampling rate is set according to the experimental requirements, and the dataset path is added.

[0067] 3c1) Read the network structure from the network structure file, initialize all network parameters, and read the hyperparameter information from the beginning of train_DDP.py.

[0068] 3c2) According to the dataset loading method set in dataset.py, load the 8-frame video blocks in each subfolder in the order set in sep_trainlist.txt. For example, frames 1 to 8 of each video are the contents of the first subfolder, frames 9 to 16 are the contents of the second subfolder, and so on. Input batchsize video blocks into the read network to obtain the corresponding reconstructed video blocks;

[0069] 3c3) Using batchsize "reconstruction-input video block pairs", update the network parameters according to the mean square error loss function and save;

[0070] 3c4) Repeat the operations in 3c2) and 3c3) multiple times until 100 trainings are completed for each subfolder in sep_trainlist.txt, and the trained network model is obtained and saved.

[0071] 3d) Create a model test file test.py in the network project folder and write the model test process in this file:

[0072] 3d1) Read the network structure from the network structure file model.py, and assign network parameters using the network model saved in 3c4);

[0073] 3d2) Convert the test video into video frames, take 8 consecutive video frames as a video block, input batchsize video blocks into the network in the order of sep_testlist.txt, and obtain the corresponding reconstructed video blocks.

[0074] 3d3) Convert the reconstructed video blocks into reconstructed video in time order.

[0075] Step 4: Train the network to obtain the network model.

[0076] Find the network training file train_DDP.py in the network project folder. This training adopts a distributed training method. Using multiple GPUs can speed up the training. Training can also be done with one GPU. Run the file, read the network structure from the network structure file model.py, and initialize the network parameters; take out the video blocks from the training set in the order in sep_trainlist.txt and input them into the network, calculate the loss between the input video block and the reconstructed video block, back-propagate the loss to obtain the network weight gradient, and use the gradient descent algorithm to update the network weight parameters. The calculation formula is:

[0077]

[0078] Where W l is the weight parameter after sequential iterations, is the weight gradient value obtained after loss back propagation, W l+1 is the updated value, and the parameters are updated multiple times after training to obtain the trained network model.

[0079] Step 5: Use the model test file test.py and the network model file model.py to obtain the reconstruction result of the test video.

[0080] 5a) Taking the video to be tested from the test data set established in step 1;

[0081] 5b) Find the model test file test.py in the project folder, run the file, read the network structure from the network structure file model.py, and assign values ​​to the network parameters using the network model trained in step 5; then convert the test video into video frames, extract the video blocks from the video frames and output the network in sequence to obtain the reconstructed video blocks;

[0082] 5c) Convert the video frames in the reconstructed video block into reconstructed video in chronological order using cv2.VideoWriter().

[0083] Example 3

[0084] The effect of the present invention can be further illustrated by the following simulation results.

[0085] The method of the present invention and the existing methods Video-MH, VCSNet and LLR-VCSNet are used to test the "Flower" in the test set, and the reconstructed video is obtained. The results are as follows: Figure 5 As shown, Figure 5 The first one is the original video frame, followed by reconstructed video frames using different methods, and the last one is the reconstructed video frame of the present invention.

[0086] Depend on Figure 5It can be seen that the reconstruction result of the present invention has a clearer structure and richer detail information, and achieves the highest scores in the picture quality evaluation indicators PSNR and SSIM, which shows that the present invention can achieve better reconstruction effect.

[0087] Depend on Figure 6 It can be seen that for each test video sequence, the score obtained by the method of the present invention is higher than that of the existing methods, and the present invention also obtains excellent reconstruction results at a lower sampling rate and has better visual effects.

[0088] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the video compressed sensing reconstruction method as described in the above embodiment 2 is implemented.

[0089] In addition, an embodiment of the present application also proposes an electronic device, wherein the terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the video compressed sensing reconstruction method as described in the above-mentioned embodiment 2 is implemented.

[0090] In addition, an embodiment of the present application also proposes a video processing system, including a video input module, a model as described in the above embodiment 1, and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs it.

[0091] Since all the technical solutions of all the aforementioned embodiments are adopted when the video compressed sensing reconstruction control program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0092] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the scope of protection of the invention.

Claims

1. A video compression sensing reconstruction model based on lightweight 3D deformable convolution, characterized in that: The model structure includes: Coding part: As the observation part, it is composed of a trainable observation matrix, which is used to compress and sample the input video frames. The observation matrix includes sampling matrices of non-key frames and key frames. Decoding part: as the reconstruction part, it is composed of a trainable reconstruction matrix, which is used to initially restore the video frames after compression sampling. The reconstruction matrix includes the reconstruction matrices of non-key frames and key frames. CNN block module: It consists of multiple convolutional layers, a pooling layer, and a deconvolution layer, and is used to further reconstruct the video frames of the decoded part output; 3D block module: used to adaptively reconstruct all video frames in the further reconstruction results output by the CNN block module using key frame information; it includes two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels and one layer of lightweight 3D deformable convolutional network with 1 convolutional kernel. The two layers of lightweight 3D deformable convolutional networks with 32 convolutional kernels are connected by multiple layers of residual connections.

2. The video compression sensing reconstruction model based on lightweight 3D deformable convolution according to claim 1 is characterized in that: The lightweight 3D deformable convolutional network is implemented by combining a 3D deformable convolution with a convolution kernel size of "3×3×1" in a spatial dimension and a 3D convolution with a convolution kernel size of "1×1×3" in a temporal dimension.

3. The video compression sensing reconstruction model based on lightweight 3D deformable convolution according to claim 1 is characterized in that: The convolution kernel size of multiple convolution layers of the CNN block module is 3×3, the step size is 1×1, the convolution is padded with zeros, and ELU is used for activation; a pooling layer of the CNN block module uses maximum pooling; a deconvolution layer of the CNN block module has a convolution kernel size of 2×2, a step size of 2×2, and no zero padding for the convolution.

4. A video compression sensing reconstruction method based on lightweight 3D deformable convolution, implemented based on the model described in any one of claims 1 to 3, characterized in that: The following steps are involved: S1: Prepare the dataset, split the video into video frames and adjust the size; S2: The video frames are divided into key frames and non-key frames, and they are sampled and reconstructed with different sampling rates and reconstruction matrices respectively; S3: The reconstruction result of each frame of video is further reconstructed through the CNN block module; S4: Combine the further reconstruction results of all frames and input them into the lightweight 3D deformable convolutional network with two layers of 32 convolution kernels of the 3D block module for overall reconstruction and output the reconstructed feature map; S5: The overall reconstructed feature map is passed through a lightweight 3D deformable convolutional network with one convolution kernel in the 3D block module to obtain the final video reconstruction result.

5. The video compression sensing reconstruction method based on lightweight 3D deformable convolution according to claim 4 is characterized in that: Step S3 is specifically as follows: the video frame first passes through multiple convolution layers, and the number of feature map channels changes from 1 to 32; then passes through the pooling layer, and the feature map size becomes smaller; then passes through multiple convolution layers, and the number of channels changes from 32 to 64; then passes through the deconvolution layer, the feature map size is restored, and the number of channels is still 64; then passes through multiple convolution layers, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained through the previous convolution layers in the channel dimension through a cross connection to obtain a 64-channel feature map; this 64-channel feature map is then passed through multiple convolution layers, and the number of feature map channels changes from 64 to 1.

6. The video compression sensing reconstruction method based on lightweight 3D deformable convolution according to claim 4 or 5, characterized in that: Steps S4 and S5 are specifically as follows: the further reconstruction results of all frames obtained in step S3 are used as five-dimensional 3D feature maps and input into the 3D block module. The five-dimensional 3D feature map of 1 channel is first passed through a lightweight 3D deformable convolution network with 32 convolution kernels to obtain a 32-channel output feature map. The 32-channel output feature map is passed through a lightweight 3D deformable convolution network with 32 convolution kernels of five layers of residual connections to obtain a 32-channel feature map. The 32-channel feature map is spliced ​​with the 32-channel feature map obtained by the first lightweight 3D deformable convolution network with 32 convolution kernels through a crossing connection in the channel dimension to obtain a 64-channel 3D feature map. The 3D feature map is passed through a layer of lightweight 3D deformable convolution network with 1 convolution kernel to obtain a one-channel output feature map, which is the reconstruction result of all frames.

7. The video compression sensing reconstruction method based on lightweight 3D deformable convolution according to claim 6 is characterized in that: Also includes: The fourth frame is extracted from the reconstruction results of all frames, and the mean square error between the reconstructed fourth frame and the fourth frame of the original video is calculated as the loss function.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the video compression sensing reconstruction method as described in any one of claims 4 to 7 is implemented.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, the video compression sensing reconstruction method as described in any one of claims 4 to 7 is implemented.

10. A video processing system, characterized in that: The system comprises a video input module, a model as claimed in any one of claims 1 to 3 and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs the video.

Citation Information

Cited By

  • Video compression system and method based on embedded platform

    CN119835440A