Video compressed sensing reconstruction method and model based on combination of 3D deformable convolution and Transform

Abstract

The invention provides a video compressed sensing reconstruction method and model based on combination of 3D deformable convolution and Transform, local features and long-distance correlation features of data can be grasped at the same time, and the quality and efficiency of video reconstruction are improved. Comprising the following steps: segmenting a video into video frames and adjusting the size; dividing a video frame into a key frame and a non-key frame, and performing sampling and reconstruction by using different sampling matrixes and reconstruction matrixes to obtain an initial reconstruction result; the initial reconstruction result is further reconstructed through a CNN independent reconstruction module; reconstructing a further reconstruction result of the key frame through a Transform module to obtain long-distance association between different areas of the key frame; further reconstruction results of all the frames and long-distance associated features obtained by the key frames through a Transform module are spliced in a channel dimension, and then feature fusion is carried out through a convolutional layer; and a result after feature fusion serves as a five-dimensional video signal to be input into a 3D deformable convolution module, and a depth reconstruction result of the video is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of video compression technology, and mainly relates to a video compression perception reconstruction method and model combining 3D deformable convolution with Transformer, which can be used to achieve video compression perception reconstruction with better visual effect. Background Art

[0002] Video compression sensing reconstruction technology aims to improve video compression efficiency while maintaining high-quality decoding results. Traditional video compression algorithms such as H.264 mainly rely on predictive coding, and there is a certain trade-off between compression efficiency and high-resolution video quality. In recent years, compressed sensing theory has provided new ideas for video compression, by utilizing the sparsity and robustness of video data, and can achieve effective recovery even after resampling.

[0003] However, existing video compression sensing reconstruction methods mainly rely on traditional convolutional neural networks (CNNs), which still have limitations in processing long-distance dependencies between video frames. Due to the complex spatiotemporal correlations between video frames, traditional CNNs are difficult to capture these high-order features, resulting in the reconstruction results not being able to fully meet the requirements in terms of video quality. Summary of the invention

[0004] The existing methods have the following shortcomings in dealing with long-distance dependencies between video frames:

[0005] 1. The traditional CNN structure is difficult to capture the complex spatiotemporal correlation between video frames; 2. The lack of an effective mechanism to learn and utilize the differences between key frames and non-key frames leads to poor spatiotemporal consistency of the reconstruction results between different frames;

[0006] The present application provides a video compression sensing reconstruction method and model combining 3D deformable convolution with Transformer, which can effectively process the long-distance dependencies between video frames, while grasping the local features and long-distance correlation features of the data, improving the performance of the model, and improving the quality and efficiency of video reconstruction.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A video compression sensing reconstruction method combining 3D deformable convolution with Transformer, comprising the following steps:

[0009] S1: Prepare the dataset, split the video into video frames and adjust the size;

[0010] S2: The video frames are divided into key frames and non-key frames. The key frames and non-key frames are sampled and reconstructed using different sampling matrices and reconstruction matrices respectively to obtain the initial reconstruction results.

[0011] S3: The initial reconstruction result of each frame is further reconstructed through a CNN independent reconstruction module;

[0012] S4: The further reconstruction results of the keyframes are reconstructed through the Transformer module to obtain the long-distance correlation between different regions of the keyframes;

[0013] S5: The further reconstruction results of all frames are concatenated with the long-distance correlation features obtained by the key frame through the Transformer module in the channel dimension, and then feature fusion is performed through a convolutional layer;

[0014] S6: The result of feature fusion is input into the 3D deformable convolution module as a five-dimensional video signal to obtain the deep reconstruction result of the video.

[0015] Furthermore, step S3 is specifically as follows: the video frame first passes through multiple convolution layers, and the number of feature map channels changes from 1 to 32; then passes through the pooling layer, and the feature map size becomes smaller; then passes through multiple convolution layers, and the number of channels changes from 32 to 64; then passes through the deconvolution layer, the feature map size is restored, and the number of channels is still 64; then passes through multiple convolution layers, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained through the previous convolution layers in the channel dimension through a cross connection to obtain a 64-channel feature map; this 64-channel feature map is then passed through multiple convolution layers, and the number of feature map channels changes from 64 to 1.

[0016] Furthermore, step S4 is specifically as follows: first, the key frame is divided into blocks, each block is converted into a vector, and long-distance feature association is realized between different vectors through Transformer Unit to obtain a reconstruction result based on block association; each pixel of the reconstruction result is transformed by a matrix to obtain the vector of each pixel, and the pixel vector is associated with long-distance features through Transformer Unit to obtain long-distance association features.

[0017] Furthermore, step S6 is specifically as follows: first, a five-dimensional 3D feature map of one channel is first passed through 32 3D deformable convolution kernels to obtain an output feature map of 32 channels, the output feature map of the 32 channels is passed through a five-layer residual connected 3D deformable convolution network to obtain a 32-channel feature map, the 32-channel feature map and the 32-channel feature map obtained by the first 3D deformable convolution are spliced ​​in the channel dimension through a cross connection to obtain a 64-channel 3D feature map; the 3D feature map is passed through three 3D deformable convolution layers to obtain an output feature map of one channel, which is the depth reconstruction result of all frames.

[0018] Furthermore, the method further includes: extracting a fourth frame from the depth reconstruction results of all frames, and calculating a mean square error between the reconstructed fourth frame and the fourth frame of the original video as a loss function.

[0019] The present application also provides a 3D deformable convolution combined with Transformer video compression sensing reconstruction model for implementing the above method, and the model structure includes:

[0020] Sampling end: As the observation part, it is composed of a trainable observation matrix, which is used to compress and sample the input video frames. The observation matrix includes sampling matrices of non-key frames and key frames.

[0021] Reconstruction end: As the reconstruction part, it is composed of a trainable reconstruction matrix, which is used to initially restore the video frames after compressive sampling. The reconstruction matrix includes the reconstruction matrices of non-key frames and key frames.

[0022] CNN independent reconstruction module: It consists of multiple convolutional layers, a pooling layer, and a deconvolution layer, and is used to further reconstruct the video frames output by the reconstruction end;

[0023] Transformer module: It consists of Transformer Unit and multiple deformation operations, which is used to obtain long-distance correlation between different regions of keyframes;

[0024] 3D deformable convolution module: It consists of 32 3D deformable convolution kernels, a five-layer residual-connected 3D deformable convolution network, and three 3D deformable convolution layers, which are used to extract video spatiotemporal information and perform deep reconstruction of the video.

[0025] Furthermore, the convolution kernel size of multiple convolution layers of the CNN independent reconstruction module is 3×3, the step size is 1×1, the convolution is padded with zeros, and ELU is used for activation; the pooling layer uses maximum pooling; the convolution kernel size of the deconvolution layer is 2×2, the step size is 2×2, and the convolution is not padded with zeros.

[0026] The present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the video compression sensing reconstruction method is implemented.

[0027] The present application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the video compression sensing reconstruction method is implemented.

[0028] The present application also provides a video processing system, including a video input module, the model and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs it.

[0029] The beneficial effects of the present invention include:

[0030] The present invention combines 3D deformable convolution with Transformer, uses Transformer to extract long-distance correlation information of key frames, uses CNN to extract local features of each frame, and fuses the long-distance correlation features established by Transformer with the local features extracted by CNN, making full use of the redundant information of the video. Each frame after the fusion of local features and long-distance correlation features is input into the residual connection 3D deformable convolution module, in which the 3D deformable convolution can fully extract the spatiotemporal information of the video, deeply reconstruct the video, and improve the overall reconstruction effect of the video.

[0031] In summary, the present invention combines 3D deformable convolution and Transformer, utilizes the advantages of both, grasps the local characteristics and long-distance correlation characteristics of the video, and uses the long-distance correlation characteristics of the key frames as a supplement to the non-key frame features. It overcomes the problem of large reconstruction quality gap between frames caused by insufficient utilization of inter-frame correlation in video compressed sensing reconstruction, and provides a video compressed sensing method that can achieve high-quality reconstruction at a low sampling rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall network block diagram of the sampling end and the reconstruction end;

[0033] Figure 2 This is the Transformer module structure diagram;

[0034] Figure 3 yes Figure 2 The Transformer Unit module structure diagram in ;

[0035] Figure 4 yes Figure 1 The structural block diagram of the CNN independent reconstruction module in;

[0036] Figure 5 yes Figure 1 The structural block diagram of the 3D deformable convolution module in;

[0037] Figure 6 This is a comparison of the reconstructed visual effects of different methods on the third frame of the Paris dataset when the sampling rate is 0.5 / 0.05;

[0038] Figure 7 This is a comparison chart of the reconstructed visual effects of different methods on the 10th frame of the dataset Flower when the sampling rate is 0.5 / 0.1. DETAILED DESCRIPTION

[0039] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] This application provides a video compression sensing reconstruction method and model combining 3D deformable convolution with Transformer. The overall implementation process is as follows:

[0042] (1) Preparation of the dataset: The videos in the dataset are segmented into individual video frames through the video segmentation function. The video frames are resized to 160×160 and grouped into 8 frames into a folder.

[0043] (2) After a set of video frames are input into the network, they are divided into key frames and non-key frames. Key frames and non-key frames are sampled and reconstructed with different sampling matrices and reconstruction matrices. Taking a key frame with a sampling rate of 0.5 as an example, the key frame of size 160×160 is divided into non-overlapping blocks of size 32×32 and transformed into a two-dimensional vector (25, 1024). This two-dimensional vector is multiplied with the trainable sampling matrix P (1024, 1024×0.5) to obtain the sampling result (25, 512). The sampling result is multiplied with the trainable reconstruction matrix R (512, 1024) to obtain (25, 1024), and then the initial reconstruction result (160, 160) is obtained after deformation.

[0044] (3) The initial reconstruction result of each frame is reconstructed through a CNN independent reconstruction module. Specifically, the CNN independent reconstruction module consists of multiple convolutional layers, a pooling layer and a deconvolution layer. Each frame will be further reconstructed through the CNN independent reconstruction module.

[0045] (4) The key frame is further reconstructed through the Transformer. Specifically, the Transformer's self-attention mechanism is used to establish long-distance associations between different regions of the key frame. First, the key frame is divided into blocks, each block is converted into a vector (token), and the long-distance feature association between different vectors is realized through the Transformer Unit to obtain a reconstruction result based on block association; each pixel of this result is transformed through a matrix to obtain the vector (token) of each pixel, and the pixel token is connected through the TransformerUnit to realize long-distance feature association, and finally a reconstruction result is obtained.

[0046] (5) The further reconstruction results of all frames are concatenated with the long-distance correlation features of the key frames obtained through the Transformer in the channel dimension, and then the features are fused through a convolutional layer to supplement the long-distance correlation features of the key frames as the non-key frame features.

[0047] (6) The results of feature fusion are combined together and input into the 3D deformable convolution module as a five-dimensional video signal. Through the five-layer residual connected 3D deformable convolution network, all frames will make full use of the rich features of the key frames for further reconstruction.

[0048] (5) The 32-channel feature map output by the five-layer residual-connected 3D deformable convolutional network is converted into a 1-channel output result after one layer of 3D deformable convolution.

[0049] The present application achieves efficient video data compression and reconstruction through an integrated hierarchical processing flow. First, the processing object is accurately prepared by cutting and resizing the video (step S1). Then, by distinguishing key frames from non-key frames and their unique sampling and reconstruction methods (step S2), the data flow is optimized to reduce the load of the overall compression process. Each frame of data is further processed by an independent reconstruction module (step S3) to maintain the visual quality of the image. The key frame introduces long-range correlation information through the Transformer module (step S4), breaking the limitations of local operations and enhancing global understanding capabilities. Subsequently, the reconstruction results of all frames are spliced ​​and fused with the long-range correlation features of the key frames in the channel dimension (step S5), which not only maintains the diversity of information, but also strengthens the correlation between frames. Finally, this integrated feature information is processed by the 3D deformable convolution module (step S6) to reshape the video signal from a three-dimensional spatial perspective, achieving highly restored and optimized deep reconstruction, and greatly improving the compression efficiency and reconstruction quality of the video. This method utilizes multimodal information processing and advanced neural network structures to significantly improve the intrinsic value and user experience of compressed videos.

[0050] Example 1

[0051] This embodiment provides a video compression sensing reconstruction method combining 3D deformable convolution with Transformer, and the implementation steps are as follows:

[0052] Step 1: Prepare the dataset.

[0053] 1a) Install Python on your computer and install the third-party libraries cv2, os and shutil under Python;

[0054] 1b) Download the 3D deformable convolution module dcn and run the bash make.sh command to generate the D3D module; and modify dcn / modules / deformable_conv.py to add the code for dimension=HWT and assign the offset in this case.

[0055] 1c) Download the UCF-101 dataset, which contains 101 human action videos with a video size of 320×240. Select the first 30 videos, the first 25 as the training set, and the last 5 as the validation set. Use cv2.videoCapture() to open the video, use cv2.ressize() to resize the video to 160×160, and then use os.makedirs() to create a folder and put every 8 frames of video in a subfolder. Name them frame_1, frame_2, etc. in the chronological order of the video frames. All subfolders constitute the training dataset and the validation dataset.

[0056] 1d) Cut the same position of each video sequence of the six standard CIF datasets "Akiyo, Coastguard, Mother_daughter, Paris, Flower, Bus" into 160×160 blocks. Since the size of the CIF dataset is 352×288, it is cut from the upper left to the upper right, and then from the lower left to the lower right, with a step length of 96, and cut into 6 blocks. Therefore, an original video of 8 frames can be cut into 6 blocks of 8 frames with a size of 160×160. Multiple subfolders after cutting are used as test sets. The overall test results of an original video of 8 frames can be obtained by splicing the test results of the blocks.

[0057] Step 2: Design a video compression sensing reconstruction model.

[0058] like Figure 1 As shown, the network includes a sampling end and a reconstruction end, the sampling end is the observation part, and the reconstruction end is the reconstruction part, where:

[0059] The observation part consists of a trainable observation matrix with a size of (B 2 , B 2 × sampling rate), where B is the block size, P and P_key are the sampling matrices of non-key frames and key frames respectively;

[0060] The reconstruction part, first of all, is a trainable reconstruction matrix with a size of (B 2 × sampling rate, B 2 ), where B is the block size, R and R_key are the reconstruction matrices of non-key frames and key frames respectively;

[0061] The reconstruction results of the key frames and non-key frames after the reconstruction matrix are further reconstructed through a CNN independent reconstruction module. The structure of the CNN independent reconstruction module is as follows: Figure 4 As shown in the figure, it consists of multiple convolutional layers, a pooling layer, and a deconvolution layer. The size of the convolution kernel is 3×3, the step size is 1×1, and the convolution is padded with 0. The convolution layer uses ElU for activation, the pooling layer uses maximum pooling, the convolution kernel size of the deconvolution layer is 2×2, the step size is 2×2, and the convolution is not padded with 0. Taking the key frame as an example, it first passes through multiple convolutional layers, and the number of convolution kernels is 1, 1, 32, 32, 32, and the number of feature map channels changes from 1 to 32; then it passes through the pooling layer, and the size of the feature map becomes smaller; then it passes through multiple convolutional layers, and the number of convolution kernels is 32, 32, 64, 64, 64, respectively. , the number of channels changes from 32 to 64; after the deconvolution layer, the number of convolution kernels is 64, the feature map size is restored, and the number of channels is still 64; after multiple convolution layers, the number of convolution kernels is 64, 64, 32, 32, 32, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained after the first five convolution layers through a cross connection in the channel dimension to obtain a 64-channel feature map; this 64-channel feature map passes through multiple convolution layers, the number of convolution kernels is 64, 64, 32, 32, 32, 1, and the number of feature map channels changes from 64 to 1.

[0062] After the above steps, each frame is further reconstructed by an independent reconstruction module;

[0063] The reconstruction result of the key frame through the reconstruction matrix is ​​input into the Transformer module for further reconstruction. The structure of the Transformer module is as follows Figure 2 As shown, it consists of a Transformer Unit and multiple deformation operations. First, the key frame is divided into blocks of 8*8 size. Each block is flattened into a one-dimensional vector (1, 64). The tokens of all vectors are input into the Transformer Unit, and finally the block-based reconstruction result is obtained through deformation. The block-based reconstruction result is further divided into 32*32 blocks. Each block is flattened into a one-dimensional vector (1024, 1). The vector is multiplied by a matrix (1, 32) to obtain the token corresponding to each pixel point. The tokens corresponding to each pixel point are input into the Transformer Unit, and finally the pixel-based reconstruction result is obtained through deformation; the reconstruction result of the key frame through the Transformer Unit and the reconstruction result of each frame through the CNN independent reconstruction module are spliced ​​in the channel dimension, and fused through a layer of convolution, so as to obtain the further reconstruction result of all frames;

[0064] The fused features of each frame are used as a five-dimensional feature map and input into the 3D deformable convolution module. First, the five-dimensional 3D feature map of one channel is passed through 32 3D deformable convolution kernels to obtain a 32-channel output feature map. The 32-channel output feature map is passed through a five-layer residual-connected 3D deformable convolution network to obtain a 32-channel feature map. The 32-channel feature map is spliced ​​with the 32-channel feature map obtained by the first 3D deformable convolution in the channel dimension through a cross connection to obtain a 64-channel 3D feature map; this feature map is passed through three 3D deformable convolution layers with the number of convolution kernels being 32, 32, and 1 respectively to obtain a one-channel output feature map, which is the reconstruction result of all frames.

[0065] The fourth frame is extracted from the reconstruction results of all frames, and the mean square error between the reconstructed fourth frame and the fourth frame of the original video is calculated as the loss function.

[0066]

[0067] MSE is the mean square error between the reconstructed frame and the original frame, y is the original video frame, and y′ is the reconstructed video frame.

[0068] Step 3, write the network project folder.

[0069] The network project folder includes: 3D deformable convolution module, Transformer module, network structure, network training and model testing files, dataset loading and preprocessing files, and hyperparameter settings. The steps are as follows:

[0070] 3a) Create a network project folder and add the 3D deformable convolution module dcn;

[0071] 3b) Design the network structure according to step 2, create a network structure file model.py in the project folder, and write the network structure in this file;

[0072] 3c) Create a network training file train_DDP.py in the network project folder, write the network training process in this file, and add hyperparameter information, batchsize = 1, and the learning rate lr is initially set to 1×10 -4 , every 10 epochs, the learning rate decays to 0.8 times of the original, and a total of 100 epochs are trained. The sampling rate is set according to the experimental requirements, and the dataset path is added.

[0073] 3c1) Read the network structure from the network structure file, initialize all network parameters, and read the hyperparameter information from the beginning of train_DDP.py.

[0074] 3c2) According to the dataset loading method set in dataset.py, load the 8-frame video blocks in each subfolder in the order set in sep_trainlist.txt. For example, frames 1 to 8 of each video are the contents of the first subfolder, frames 9 to 16 are the contents of the second subfolder, and so on. Input batchsize video blocks into the read network to obtain the corresponding reconstructed video blocks;

[0075] 3c3) Using batchsize "reconstruction-input video block pairs", update the network parameters according to the mean square error loss function and save;

[0076] 3c4) Repeat the operations in 3c2) and 3c3) multiple times until 100 trainings are completed for each subfolder in sep_trainlist.txt, and the trained network model is obtained and saved.

[0077] 3e) Create a model test file test.py in the network project folder and write the model test process in this file:

[0078] 3e1) Read the network structure from the network structure file model.py, and assign network parameters using the network model saved in 4d4);

[0079] 3e2) Convert the test video into video frames, take 8 consecutive video frames as a video block, and input batchsize video blocks into the network in the order of sep_testlist.txt to obtain the corresponding reconstructed video blocks.

[0080] 3e3) Convert the reconstructed video blocks into reconstructed video in time order.

[0081] Step 4: Train the network to obtain the network model.

[0082] Find the network training file train_DDP.py in the network project folder. This training adopts a distributed training method. Using multiple GPUs can speed up the training. You can also use a single GPU for training. Run this file, read the network structure from the network structure file model.py, and initialize the network parameters; take out the video blocks from the training set in the order in sep_trainlist.txt and input them into the network, calculate the loss between the input video block and the reconstructed video block, back-propagate the loss to obtain the network weight gradient, and use the gradient descent algorithm to update the network weight parameters. The calculation formula is:

[0083]

[0084] Where W l is the weight parameter after sequential iterations, is the weight gradient value obtained after loss back propagation, W l+1 is the updated value, and the parameters are updated multiple times after training to obtain the trained network model.

[0085] Step 5: Use the model test file test.py and the network model file model.py to obtain the reconstruction result of the test video.

[0086] 5a) Taking the video to be tested from the test data set established in step 1;

[0087] 5b) Find the model test file test.py in the project folder, run the file, read the network structure from the network structure file model.py, and assign values ​​to the network parameters using the network model trained in step 4; then convert the test video into video frames, extract the video blocks from the video frames and output the network in sequence to obtain the reconstructed video blocks;

[0088] 5c) Convert the video frames in the reconstructed video block into reconstructed video in chronological order using cv2.VideoWriter().

[0089] Example 2

[0090] The effect of the present invention can be further illustrated by the following simulation results.

[0091] Simulation 1: The method of the present invention and the existing methods Video-MH, RRS, VCSNet and LLR-VCSNet are used to test "Paris" and "FLower" in the test set, respectively, and the reconstructed videos are obtained. The results are shown in FIG. Figure 6 and Figure 7 As shown, the first picture in the figure is the original video frame, the following pictures are reconstructed video frames using different methods, and the last picture is the reconstructed video frame of the present invention.

[0092] Depend on Figure 6 and Figure 7 It can be seen that under the same sampling rate, the reconstruction result of the present invention has a clearer structure and richer detail information, and achieves the highest score in the picture quality evaluation indicators PSNR and SSIM, which shows that the present invention can achieve better reconstruction effect.

[0093] Depend on Figure 6 and Figure 7 It can be seen that for the test video sequence, the scores obtained by the method of the present invention are higher than those of the existing methods, and the visual effect of the reconstructed video frame of the present invention is still good when the key frame sampling rate remains unchanged and the non-key frame sampling rate is reduced.

[0094] In addition, an embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the video compressed sensing reconstruction method as described in the above-mentioned embodiment 1 is implemented.

[0095] In addition, an embodiment of the present application also proposes an electronic device, wherein the terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the video compressed sensing reconstruction method as described in the above-mentioned embodiment 1 is implemented.

[0096] In addition, an embodiment of the present application also proposes a video processing system, including a video input module, a model as described in the above embodiment 1, and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs it.

[0097] Since all the technical solutions of all the aforementioned embodiments are adopted when the video compressed sensing reconstruction control program is executed by the processor, it has at least all the beneficial effects brought by all the technical solutions of all the aforementioned embodiments, which will not be described one by one here.

[0098] Obviously, the above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A video compression sensing reconstruction method combining 3D deformable convolution with Transformer, characterized in that: The following steps are involved: S1: Prepare the dataset, split the video into video frames and adjust the size; S2: The video frames are divided into key frames and non-key frames. The key frames and non-key frames are sampled and reconstructed using different sampling matrices and reconstruction matrices respectively to obtain the initial reconstruction results. S3: The initial reconstruction result of each frame is further reconstructed through a CNN independent reconstruction module; S4: The further reconstruction results of the keyframes are reconstructed through the Transformer module to obtain the long-distance correlation between different regions of the keyframes; S5: The further reconstruction results of all frames are concatenated with the long-distance correlation features obtained by the key frame through the Transformer module in the channel dimension, and then feature fusion is performed through a convolutional layer; S6: The result of feature fusion is input into the 3D deformable convolution module as a five-dimensional video signal to obtain the deep reconstruction result of the video.

2. According to the 3D deformable convolution combined with Transformer video compression sensing reconstruction method of claim 1, it is characterized in that: Step S3 is specifically as follows: the video frame first passes through multiple convolution layers, and the number of feature map channels changes from 1 to 32; then passes through the pooling layer, and the feature map size becomes smaller; then passes through multiple convolution layers, and the number of channels changes from 32 to 64; then passes through the deconvolution layer, the feature map size is restored, and the number of channels is still 64; then passes through multiple convolution layers, and the number of feature map channels changes from 64 to 32; this 32-channel feature map is spliced ​​with the 32-channel feature map obtained through the previous convolution layers in the channel dimension through a cross connection to obtain a 64-channel feature map; this 64-channel feature map is then passed through multiple convolution layers, and the number of feature map channels changes from 64 to 1.

3. The video compression sensing reconstruction method of 3D deformable convolution combined with Transformer according to claim 1 or 2 is characterized in that: Step S4 is specifically as follows: first, the key frame is divided into blocks, each block is converted into a vector, and long-distance feature association is realized between different vectors through Transformer Unit to obtain a reconstruction result based on block association; each pixel of the reconstruction result is transformed by a matrix to obtain the vector of each pixel, and the pixel vector is associated with long-distance features through Transformer Unit to obtain long-distance association features.

4. The video compression sensing reconstruction method of 3D deformable convolution combined with Transformer according to claim 3 is characterized in that: Step S6 is specifically as follows: first, the five-dimensional 3D feature map of one channel is first passed through 32 3D deformable convolution kernels to obtain an output feature map of 32 channels, the output feature map of the 32 channels is passed through a five-layer residual connected 3D deformable convolution network to obtain a 32-channel feature map, the 32-channel feature map and the 32-channel feature map obtained by the first 3D deformable convolution are spliced ​​in the channel dimension through a cross connection to obtain a 64-channel 3D feature map; this 3D feature map is passed through three 3D deformable convolution layers to obtain an output feature map of one channel, which is the depth reconstruction result of all frames.

5. The video compression sensing reconstruction method of 3D deformable convolution combined with Transformer according to claim 4 is characterized in that: Also includes: The fourth frame is extracted from the depth reconstruction results of all frames, and the mean square error between the reconstructed fourth frame and the fourth frame of the original video is calculated as the loss function.

6. A video compression sensing reconstruction model combining 3D deformable convolution and Transformer, used to implement the method according to any one of claims 1 to 5, characterized in that: The model structure includes: Sampling end: As the observation part, it is composed of a trainable observation matrix, which is used to compress and sample the input video frames. The observation matrix includes sampling matrices of non-key frames and key frames. Reconstruction end: As the reconstruction part, it is composed of a trainable reconstruction matrix, which is used to initially restore the video frames after compressive sampling. The reconstruction matrix includes the reconstruction matrices of non-key frames and key frames. CNN independent reconstruction module: It consists of multiple convolutional layers, a pooling layer, and a deconvolution layer, and is used to further reconstruct the video frames output by the reconstruction end; Transformer module: It consists of Transformer Unit and multiple deformation operations, which are used to obtain long-distance correlation between different regions of keyframes; 3D deformable convolution module: It consists of 32 3D deformable convolution kernels, a five-layer residual-connected 3D deformable convolution network, and three 3D deformable convolution layers, which are used to extract video spatiotemporal information and perform deep reconstruction of the video.

7. The video compression sensing reconstruction model of 3D deformable convolution combined with Transformer according to claim 6 is characterized in that: The convolution kernel size of multiple convolution layers of the CNN independent reconstruction module is 3×3, the step size is 1×1, the convolution is padded with zeros, and ELU is used for activation; the pooling layer uses maximum pooling; the convolution kernel size of the deconvolution layer is 2×2, the step size is 2×2, and the convolution is not padded with zeros.

8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the video compression sensing reconstruction method as described in any one of claims 1 to 5 is implemented.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, the video compression sensing reconstruction method according to any one of claims 1 to 5 is implemented.

10. A video processing system, characterized in that: The system comprises a video input module, a model as described in any one of claims 6 to 7, and a video output module. The system performs compressed sensing reconstruction on the input video through the model and outputs the video.

Images