Video high space-time resolution signal processing method

Abstract

The application discloses a video high space-time resolution signal processing method, which adopts a signal and information processing algorithm to restore and reconstruct a high space-time resolution video sequence, namely a video super-resolution reconstruction method.The steps are as follows: taking frame sequences of a video in sequence; starting from the third frame of the video, motion estimation is performed on each frame and the two frames before and after the frame by using an optical flow method; four pairs of motion estimation images and an intermediate frame are combined into a high-dimensional image block of five images; an OF depth convolution super-resolution network is constructed, a shallow network extracts image information, and a last sub-pixel convolution layer performs reconstruction of a super-resolution image; the high-dimensional image block is sent into the depth convolution network for training; and finally, a video frame with low quality or low resolution is sent into the network for reconstruction.The application has the advantages of good reconstruction quality, fast reconstruction speed, good reconstruction effect compared with a traditional video super-resolution model, and can realize real-time video reconstruction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing and deep learning application, and particularly relates to a video high space-time resolution signal processing method combining optical flow method and deep network. BACKGROUND

[0002] Video often has the problem of resolution reduction due to high-frequency signal loss in the process of shooting, transmission and saving. Video super-resolution reconstruction is an effective video signal processing method for directly obtaining high-resolution images from low-resolution video frames. Video super-resolution reconstruction can be applied to monitoring, video recording, high-definition video and television broadcasting, etc.

[0003] The current mainstream super-resolution reconstruction methods include interpolation method, reconstruction method and learning method. The interpolation-based method has simple principle and small amount of calculation. The reconstruction-based method inversely uses the degradation model of the image. The core idea of the method is modeling, and a corresponding mathematical model is formed between low resolution and high resolution. The learning-based super-resolution reconstruction algorithm learns the corresponding relationship between high and low resolution images more deeply. This kind of algorithm reconstructs the super-resolution image through three stages of feature extraction, learning image features and super-resolution reconstruction.

[0004] The end-to-end model feature of deep learning enables deep learning to be effectively applied to super-resolution reconstruction. SRCNN (Super-resolution Reconstruction Neural Networks) network applies deep learning to super-resolution reconstruction, and improves the super-resolution reconstruction quality, but the structure is simple, and the reconstruction quality and speed are general. ESPCN (Efficient Sub-Pixel Convolutional Neural Network) introduces the idea of sub-pixel convolution layer, and designs a three-layer network structure, which greatly improves the reconstruction efficiency. Although it can be directly applied to video reconstruction, the network structure is too simple, and the inter-frame correlation and motion information of the video are not considered. SUMMARY

[0005] The purpose of the application is to perform super-resolution reconstruction on the video for the problem of high-frequency signal loss and resolution loss of the video, to improve the video reconstruction quality, and to reconstruct high frame rate and high-definition video in real time. Therefore, a video high space-time resolution signal processing method combining optical flow method and deep network is proposed.

[0006] A video high space-time resolution signal processing method combining optical flow method and deep network comprises the following steps:

[0007] Step 1, extracting the frame sequence of the video file in order;

[0008] Step 2, starting from the 3rd frame of the video, each frame is motion estimated with its two preceding and succeeding frames using optical flow method; the resulting 4 pairs of motion estimation images are combined with the intermediate frame into a high-dimensional image block of 5 images;

[0009] Step 3, constructing an OF depth convolution super-resolution network, the shallow network extracts image information, and the last sub-pixel convolution layer reconstructs the super-resolution image;

[0010] Step 4, sending the high-dimensional image block into the depth convolution network for training;

[0011] Step 5, sending the degraded or low-resolution video frame into the network for super-resolution image reconstruction.

[0012] Further, the high-dimensional image block construction method in Step 2 is as follows:

[0013] Step 2-1, selecting 5 video frames in sequence, we mark the frame as the nth frame, then the two preceding frames are n-2 and n-1 frames, and the two succeeding frames are n+1 and n+2 frames;

[0014] Step 2-2, before inputting into the neural network, motion estimation is performed on the five images, taking the central frame n as the reference frame, and using optical flow method to perform motion estimation on the remaining four frames to generate four motion estimation images;

[0015] Step 2-3, combining the four motion-estimated frame images with the reference frame into a high-dimensional image block of 5 images.

[0016] Further, the reference frame taken in Step 2-1 needs to be within the range of the 3rd to n-2th frame in the video frame sequence, so as to ensure the generation of four motion estimation images before and after.

[0017] Further, the depth convolution super-resolution network model constructed in Step 3 is a 4-layer structure, divided into three convolution layers and one sub-pixel convolution layer, and the input image is a high-dimensional image block of 15 channels. The first layer of convolution uses a 5x5 convolution kernel with a total of 64, and the output is a 64-channel feature image after convolution; the second layer uses a 3x3 convolution kernel with a total of 32, which convolves the 64-channel feature image of the previous layer, and the output image is 32 channels; the third layer uses a 3x3 convolution kernel with a total of 3xr 2 feature images. 2

[0018] Further, the construction method of the depth convolution network for video super-resolution reconstruction in Step 3 is as follows:

[0019] Step 3-1, since a high-dimensional image block of 5 images is generated, the input high-dimensional image has a dimension of HxWx5xC; ​

[0020] Step 3-2, the shallow network uses convolution operation to extract image feature information;

[0021] Step 3-3, the second-to-last layer of the shallow network forms HxWxCxr 2 image feature block;

[0022] Step 3-4, the last layer of the network sub-pixel convolution layer reconstructs a high-resolution image from the HxWxCxr 2 image feature block.

[0023] Further, the specific implementation method of the sub-pixel convolution layer is to use a convolution kernel W of size S K S with a step size of 1 / r to activate different parts of W S , and the weights between pixels do not need to be calculated. The activated "pixels" depend on their positions, and at most weights are activated; when the convolution kernel is gradually scanned through the entire feature image space, different sub-pixels are periodically activated according to their positions. This periodic method is realized by mod(x, r) and mod(y, r), x and y are output pixels corresponding to the high-resolution space;

[0024] The above process is represented by the formula

[0025]

[0026] In the above formula, I SR is the reconstructed super-resolution image, I LR is the low-resolution image block output by the last layer, PS is the periodic activation function, W L is the width of the low-resolution image, and b L is the bias of the neural network. The mean square error is used as the basis for network iteration, and the formula is as follows:

[0027]

[0028] In the above formula, is the error in iteration, r is the scaling factor, H and W respectively indicate the height and width of the image, I LR and I HR are the input low-resolution image and the original high-resolution image, respectively.

[0029] Further, the training step in step 4 is: step 4-1, using Python language, Microsoft's open source deep learning framework TensorFlow framework, batch-size is 32, epoch is 100, sub-image size is 17*17, learning rate is 0.001; step 4-2, the network is trained, and part of the video sequence of the Xiph video training data set constructed by the Xiph Foundation is selected, wherein 10 video sequences contain 2695 image frames; the high-resolution image of the training data set and the high-dimensional image block based on the frame are respectively taken as the label and the input data and sent into the network for training.

[0030] The beneficial effects of the present application are: (1) considering the signal correlation between video frames, introducing the inter-frame motion estimation of video into the video super-resolution reconstruction network, which can effectively improve the reconstruction quality of video frames and reduce the blurring effect of reconstructed video frames. (2) The network structure introduces a sub-pixel convolution layer, which does not need to be enlarged by interpolation before reconstructing the low-resolution video frame, which can effectively improve the reconstruction efficiency and achieve real-time video reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a schematic diagram of the processing method framework of the present application. EMBODIMENT

[0032] The technical solutions of the present application will be further described in detail below in combination with the drawings of the specification.

[0033] A video high space-time resolution signal processing method combining optical flow method and deep network includes the following steps: step 1, extracting the frame sequence of the video file in order.

[0034] Step 2, starting from the 3rd frame of the video, each frame is motion estimated with the two frames before and after it by optical flow method; and the generated four motion estimation images and the middle frame are combined into a high-dimensional image block of five images. The high-dimensional image block construction method in step 2 is: step 2-1, selecting five continuous video frames from the video frame sequence, we mark the frame as the nth frame, then the two frames before are n-2 and n-1 frames, and the two frames after are n+1 and n+2 frames. Step 2-2, before inputting into the neural network, motion estimation is performed on the five images, taking the center frame n as the reference frame, and using optical flow method to perform motion estimation on the remaining four frames to generate four motion estimation images.

[0035] Step 2-3, combining the four motion-estimated frame images and the reference frame into a high-dimensional image block of five images.

[0036] The reference frame taken in step 2-1 needs to be within the range of the 3rd to the n-2th frame of the video frame sequence, so as to ensure the generation of four motion estimation images before and after.

[0037] Step 3, build OF depth convolution super-resolution network, shallow network extracts image information, and the last layer sub-pixel convolution layer reconstructs super-resolution image.

[0038] The depth convolution super-resolution network model built in step 3 is a 4-layer structure, which is divided into three convolution layers and one sub-pixel convolution layer, and the input image is a high-dimensional image block of 15 channels. The first layer of convolution uses 64 convolution kernels of 5x5, and the output is a 64-channel feature image after convolution; the second layer uses 32 convolution kernels of 3x3, and the output image is 32 channels after convolution of the 64-channel feature image of the previous layer; the third layer uses 3xr 2 convolution kernels of 3x3, and the convolution output is a 3xr 2 feature image.

[0039] The construction method of the depth convolution network for video super-resolution reconstruction in step 3 is as follows:

[0040] Step 3-1, since 5 high-dimensional image blocks are generated, the input high-dimensional image has a dimension of HxWx5xC.

[0041] Step 3-2, the shallow network uses convolution operation to extract image feature information.

[0042] Step 3-3, the last but one layer of shallow network forms an image feature block of HxWxCxr 2 .

[0043] Step 3-4, the last layer network sub-pixel convolution layer reconstructs a high-resolution image from the image feature block of HxWxCxr 2 .

[0044] The specific implementation method of the sub-pixel convolution layer is to use a convolution kernel of size KS to perform convolution with a step of 1 / r, which can activate different parts of Ws, and the weight between pixels does not need to be calculated. The activated "pixels" depend on their positions, and at most r weights are activated; when the convolution kernel scans the entire feature image space step by step, different sub-pixels are activated periodically according to their positions, and this periodic method is realized by mod(x, r) and mod(y, r), x and y are output pixels corresponding to the high-resolution space.

[0045] The above process can be expressed by the formula

[0046]

[0047] In the formula, I SR is the reconstructed super-resolution image, I LR is the low-resolution image block output by the last layer, PS is the periodic activation function, and WL is the width of the low-resolution image, b L is the bias of the neural network. The mean square error is used as the basis for network iteration, and the formula is as follows:

[0048]

[0049] In the above formula, is the error in iteration, r is a scaling factor, H and W respectively indicate the height and width of the input image, I LR and I HR are the input low-resolution image and the original high-resolution image respectively.

[0050] Step 4, the high-dimensional image block is sent into the deep convolutional network for training. The training steps in step 4 are: step 4-1, using Python language, Microsoft's open source deep learning framework TensorFlow framework, batch-size is 32, epoch is 100, sub-image size is 17x17, learning rate is 0.001.

[0051] Step 4-2, the network is trained, and part of the video sequence of the Xiph Foundation constructed video training data set Xiph is selected, which contains 10 video sequences and a total of 2695 image frames; the high-resolution image of the training data set and the high-dimensional image block based on the frame are respectively sent into the network as the label and the input data for training.

[0052] Step 5, the degraded or low-resolution video frame is sent into the network for super-resolution image reconstruction.

Claims

1. A video high-temporal-spatial resolution signal processing method, which is a video high-temporal-spatial resolution signal processing method combining optical flow method and deep network, characterized in that: The method comprises the following steps: Step 1, extracting the frame sequence of the video file in order; Step 2, starting from the 3rd frame of the video, each frame is subjected to optical flow motion estimation with the two frames before and after it; the generated four motion estimation images are combined with the intermediate frame into a high-dimensional image block of five images; Step 3, constructing an OF deep convolutional super-resolution network, the shallow network extracts image information, and the last sub-pixel convolutional layer performs reconstruction of the super-resolution image; The OF depth convolution super-resolution network constructed in step 3 is a 4-layer structure, which is divided into three convolution layers and one sub-pixel convolution layer, and the input image is a high-dimensional image block with 15 channels; the first layer of convolution uses 64 convolution kernels with a size of 5*5, and outputs a feature image with 64 channels after convolution; the second layer uses 32 convolution kernels with a size of 3*3, convolves the 64-channel feature image of the previous layer, and outputs an image with 32 channels; the third layer uses 3*r 2 convolution kernels with a size of 3*3, and outputs a 3*r 2 feature image after convolution; Step 4, sending the high-dimensional image block into the OF deep convolutional super-resolution network for training; Step 5, sending the degraded video frame into the OF deep convolutional super-resolution network for super-resolution image reconstruction; The high-dimensional image block construction method in step 2 is as follows: Step 2-1, selecting five video frames in sequence, and marking the intermediate frame as the nth frame, then the two frames before are n-2 and n-1 frames, and the two frames after are n+1 and n+2 frames; Step 2-2, before inputting into the neural network, the five images are subjected to motion estimation, taking the central frame n as the reference frame, and using the optical flow method to estimate the motion of the remaining four frames to generate four motion estimation images; Step 2-3, combining the four motion-estimated frame images with the reference frame into a high-dimensional image block of five images; The training step in step 4 is as follows: Step 4-1, using Python language, Microsoft's open-source deep learning framework TensorFlow framework, batch-size is 32, epoch is 100, sub-image size is 17x17, and learning rate is 0.001; Step 4-2, training the network, selecting part of the video sequence of the Xiph video training dataset constructed by the Xiph Foundation, which contains 10 video sequences and a total of 2695 image frames; the high-resolution image of the training dataset and the high-dimensional image block of the high-resolution image are sent into the network as labels and input data respectively for training.

2. The method of video high spatio-temporal resolution signal processing according to claim 1, characterized in that: The reference frame taken in step 2-1 needs to be within the range of the 3rd to n-2th frame of the video frame sequence, so as to ensure the generation of four motion estimation images before and after.

3. The method of video high-temporal resolution signal processing according to claim 1, characterized in that: The construction method of the OF deep convolutional super-resolution network in step 3 is as follows: Step 3-1, since a high-dimensional image block of five images is generated, the input high-dimensional image has a dimension of HxWx5xC; Step 3-2, the shallow network uses convolution operation to extract image feature information; Step 3-3, the second to last layer shallow network forms HxWxCxr 2 of image feature blocks; Step 3-4, the last network sub-pixel convolution layer from HxWxCxr 2 of image features The high-resolution image is reconstructed in the block.

4. The method of video high-temporal resolution signal processing according to claim 3, characterized in that: The specific implementation method of the sub-pixel convolution layer is to use a convolution kernel W S with a size of K S to perform a convolution activation W S with a step of 1 / r. The weights between pixels do not need to be calculated, and the activated pixels depend on the positions of the pixels, and at most K weights are activated. When the convolution kernel scans the entire feature image space step by step, different sub-pixels are activated periodically according to the positions of the different sub-pixels. This periodic method is implemented by mod(x, r) and mod(y, r), x and y being output pixels corresponding to the high-resolution space. The formula is as follows: ； where I SR is the reconstructed super-resolution image, I LR is the last layer output of the low-resolution image block, PS is a periodic activation function, W L is the width of the low-resolution image, b L is the bias of the neural network; The mean square error is used as the basis for network iteration, and the formula is as follows: ； In the above equation, is the error in the iteration, r is a scaling factor, H and W denote the height and width of the image, respectively, I LR and I HR are the input low resolution image and the original high resolution image, respectively.

Images