An adaptive frame interpolation method based on spatiotemporal attention mechanism

Abstract

The present invention discloses an adaptive frame insertion method based on a spatiotemporal attention mechanism, and relates to the field of video processing; by adding a spatiotemporal attention mechanism to an encoder-decoder structure, the feature extraction of adjacent frames is ensured to be complete. The present invention uses a U-Net network with spatiotemporal attention to extract the features of an image, and focuses on learning the features of the image from the channel and spatial dimensions, respectively, rather than treating different channel and spatial dimension information with the same standard, so that the feature extraction is complete, which is helpful for the parameter generation of the later sub-network; the boundary filling method used for the input adjacent frames avoids the generation of distorted synthetic frames due to occlusion and other reasons during the generation process.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to an adaptive frame interpolation method based on a spatiotemporal attention mechanism. Background Art

[0002] With the popularity of electronic playback devices, more and more people care about the viewing experience of videos, so it is very important to improve the temporal resolution of videos. Videos with low temporal resolution will first give people a bad visual experience, and at the same time, it will increase the difficulty of later processing of videos, such as improving video quality and video. Video interpolation does not require the use of any equipment or other external forces to obtain slow-motion videos, improve temporal resolution, and bring better visual experience and improve visual smoothness. Unlike the pixel difference of pictures, because videos also have the concept of timeline, it is necessary to understand the overall movement of the video. Since movement in real life is too complicated, the estimation of movement provides great difficulty for video interpolation.

[0003] Existing methods include traditional video frame interpolation methods and deep learning-based video frame interpolation methods.

[0004] The traditional video frame interpolation method uses frame sampling, frame blending, and motion compensation to interpolate frames. The implementation step of frame sampling is to use the current frame as the compensation frame. Its essence is to lengthen the display time of each frame. In theory, it is equivalent to not interpolating frames, but only increases people's visual viewing experience. The disadvantage of this method is that it causes the overall video picture to be unsmooth. The implementation step of frame blending is to increase the transparency of two adjacent frames, and then blend them into an intermediate frame, which has the effect of video interpolation. Its disadvantage is that there will be blur between two frames, and the effect is not good. Motion compensation is the most commonly used method in frame interpolation. Its implementation principle is to find the motion trend of the full frame for the offset value between two frames in the x and y directions, and then obtain the intermediate synthetic frame through certain calculations. Its disadvantage is that first of all, it only considers the x and y directions. For the foreground target with motion in the video and the background in the video is too complex, distortion and artifacts will occur. In general, the traditional method is too simple in structure and steps, resulting in distortion and blur in the final generated video. At the same time, it cannot process complex videos, and will cause the final video to play unsmoothly, with freezes, and the edges of moving objects to become blurred.

[0005] Video frame interpolation method based on deep learning: Generate intermediate frames by training models. Usually, most methods process motion estimation and pixel synthesis separately, or directly merge these two steps together to obtain intermediate frames. That is, the synthesis of intermediate frames is regarded as a local convolution operation of two adjacent original frames. The correlation coefficient of local motion and pixel synthesis between two adjacent original frames is obtained through the convolution kernel. Since the motion in real life is often complex, this requires an operation, that is, referencing any number of pixels at any position in the input frame. Super SloMo of Huaizu Jiang's team predicts the optical flow from the intermediate frame to the adjacent frames and obtains the weights of the synthesized frame to the previous and next frames. At the same time, it also uses a visual map to indicate whether the pixels of the previous and next frames appear in the synthesized frame, and uses two U-Net structures to generate intermediate synthetic frames. MEMC-Net of Wenbo Bao's team starts with motion estimation and compensation drive, and designs an adaptive deformation layer to integrate optical flow, interpolation kernel technology and U-Net to synthesize intermediate frames. AdaCoF of Hyeongmin Lee's team synthesizes intermediate frames by estimating the kernel weights and offset vectors of all target pixels one by one. The U-Net structure is also used for the synthesis of the target frame.

[0006] The implementation effect of deep learning frame difference method is better than that of traditional methods in most cases, and it has more application scenarios. The traditional deep learning video frame interpolation method uses U-Net network to generate the final synthetic frame. The advantage of this encoder-decoder structure is that it is relatively simple to implement, but the disadvantage is that due to the overly simple structure, the feature extraction of each frame is incomplete at the encoder stage, which will also cause deformation and distortion of the frames generated later.

[0007] In summary, existing methods will reduce the effect of frame interpolation in complex videos with moving foreground objects or occlusions in the video, resulting in poor results in generating intermediate frames. Summary of the invention

[0008] In order to solve the shortcomings of the prior art, the present invention provides an adaptive frame interpolation method based on spatiotemporal attention, which ensures that the features of adjacent frames are completely extracted by adding a spatiotemporal attention mechanism to the encoder-decoder structure. At the same time, the problem of occlusion in adjacent frames is solved by edge filling.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] An adaptive frame interpolation method based on spatiotemporal attention mechanism includes the following steps:

[0011] S1: Input two consecutive frames I1 and I2, and perform edge filling operation on the input frames I1 and I2, wherein the edge filling operation is to fill x pixels in width and height dimensions respectively;

[0012] S2: Merge the color channels of I1` and I2` after edge filling to form 6 color channels, and input them into the Attention U-Net network;

[0013] Build a frame insertion model, including a U-Net network with an attention mechanism, a sub-network for generating adaptive parameters, and a generation network;

[0014] The feature extraction of I1` and I2` is performed through the Attention U-Net generation network with a spatiotemporal attention mechanism, and the sub-network generates adaptive parameters based on the feature estimation extracted by the Attention U-Net; the generation network synthesizes the intermediate frame based on the adaptive parameters generated by the sub-network and the input frame; the sub-network generates adaptive parameters based on the features obtained by the Attention U-Net, and the adaptive parameters include parameters w1, α1 and β1 of the first frame, and parameters w2, α2 and β2 of the second frame, wherein w1 and w2 represent the weight of the first frame and the weight of the second frame respectively, α1 and α2 are the offsets added by the variable convolution function in the x direction of the first frame and the second frame, β1 and β2 represent the offsets added by the variable convolution function in the y direction of the first frame and the second frame, and the adaptive parameters are all greater than 0;

[0015] The Attention U-Net network includes an attention mechanism, an encoder and a decoder;

[0016] The attention mechanism adopts a hybrid attention mechanism structure, which is integrated by a channel attention module and a spatial attention module, and jointly learns the features of the input frame from the channel and spatial dimensions respectively;

[0017] The hybrid Attention introduced by the encoder consists of a convolutional layer, a reLU function, and an average downsampling layer in the encoder.

[0018] S3: backward convolution C of I1` b Operation plus I2` forward convolution C f Operation I 合成 , for C f and C b The operation is to use adaptive operation, as shown in formula (1):

[0019] I 合成 =C f (I′1)+C b (I′2) (1)

[0020] S4: Set masks for adjacent frames I1` and I2`. The mask matrix of I1` is A, the mask of the filling part is 0, and the mask that appears in I1` but not in I2` is 1, and the mask that does not appear is 0; the mask matrix of I2` is B, and the mask of the filling part is 0, and the mask that appears in I2` but not in I1` is 1, and the mask that does not appear is 0, as shown in formula (2):

[0021] I 合成 =A*C f (I′1)+B*C b (I′2) (2)

[0022] S5: Generate an intermediate frame, using different weights for different directions, different pixels, and distances from the reference pixel, using the optical flow and pixel value of each pixel, as shown in formula (3):

[0023]

[0024] Among them, W*H is the resolution of the adjacent frames after padding, w is the weight of each reference pixel, α and β are offsets. Since the displacement of each pixel is not necessarily an integer number of pixels, the frame interpolation model is used to learn α and β, and d is the distance coefficient from each pixel to the reference pixel.

[0025] The loss function in the frame interpolation model calculates the loss between the synthetic frame and the groundtruth, connects the generated frame and one of the input frames in chronological order, and uses a discriminator to distinguish the generated frame from one of the input frames.

[0026] The beneficial effects of adopting the above technical solution are:

[0027] 1. The present invention uses a U-Net network with spatiotemporal Attention to extract the features of the image, and focuses on learning the features of the image from the channel and spatial dimensions respectively, instead of treating different channel and spatial dimension information with the same standard, so that the feature extraction is complete and helpful for the subsequent sub-network generation.

[0028] 2. The border filling method is used for the adjacent input frames to avoid the generation of distorted synthetic frames due to occlusion and other reasons during the generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A flow chart of an adaptive frame interpolation method based on a spatiotemporal attention mechanism provided by an embodiment of the present invention;

[0030] Figure 2 A structural diagram of a frame insertion model provided by an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of the effect of a synthesized frame obtained based on adjacent frames according to an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of the effects obtained by training different epochs provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0034] like Figure 1 As shown, the method of this embodiment is described as follows.

[0035] An adaptive frame interpolation method based on spatiotemporal attention mechanism includes the following steps:

[0036] S1: Input two consecutive frames I1 and I2, such as Figure 3 As shown in I1 and I2, Figure 3 I1 and I2 in the above table represent the input scores of adjacent frames, I 合成 Indicates the synthetic frame generated according to the model of this patent. Considering that there may be some object occlusion between two frames and the existence of the time axis between two frames causes the object in the previous frame to appear but disappear in the next frame, or not appear in the previous frame but just appear in the next frame, for example Figure 3 In the figure, the ball in the person's hand is rotating, so there will be an occlusion surface as it rotates. The input frames I1 and I2 are edge-filled. The edge-filling operation is to fill x pixels in the width and height dimensions. This patent fills 80 pixels in the up, down, left, and right directions of the input frame.

[0037] S2: Merge the RGB color channels of I1` and I2` after edge filling to form 6RGB channels, and input them into the Attention U-Net network to extract features, such as Figure 2 The Encoder-Decoder part is shown in the yellow box.

[0038] Construct a frame insertion model, including a U-Net network with an attention mechanism, a sub-network for generating adaptive parameters, and a generating network. The structure of the frame insertion model is as follows: Figure 2 shown.

[0039] First, the Attention U-Net network is used to extract features of I1` and I2`. Then, the sub-network estimates the adaptive parameters required by the final generation network based on the features extracted by the Attention U-Net. The generation network synthesizes the intermediate frame based on the parameters generated by the sub-network and the input frame. The adaptive parameters generated by the features obtained by the network include the parameters w1, α1 and β1 of the first frame, and the parameters w2, α2 and β2 of the second frame. These generation parameters are all greater than 0, such as Figure 2 shown.

[0040] The Attention U-Net network includes an attention mechanism, an encoder and a decoder;

[0041] The attention mechanism adopts a hybrid attention mechanism structure, which is integrated by a channel attention module and a spatial attention module, which jointly learns the features of the picture in a targeted manner from the channel and spatial dimensions respectively, which plays a role in emphasizing or suppressing information and also helps the flow of information within the network. The U-Net network structure is relatively simple and easy to implement, and it can make the overall network lighter. First, the convolutional network with Attention can be directly end-to-end learned. Secondly, it will automatically focus on the area of ​​the video frame with features during the network forward transmission. Finally, it will not introduce a large number of parameters and calculations;

[0042] The mixed spatiotemporal Attention introduced by the encoder, each module in the encoder is composed of a convolutional layer, a ReLU function and an average downsampling layer, and each module in the decoder is composed of a convolutional layer, a ReLU function and an upsampling layer.

[0043] The module for generating w in the subnetwork for generating adaptive parameters w, α and β is composed of a mixture of convolutional layer, ReLU function, average downsampling layer and softmax function. The module for generating α and β is composed of a mixture of convolutional layer, ReLU function and average downsampling layer.

[0044] S3: backward convolution C of I1` b Operation plus I2` forward convolution C f Operation I 合成 , for C f and C b The operation uses adaptive operation. The generation of target pixel points needs to take into account the reference pixels in any direction and any position of the previous and next frames. Therefore, an adaptive synthetic kernel convolution will be generated here according to each target pixel point in any method and at any position.

[0045] I 合成 =C f (I′1)+C b (I′2) (1)

[0046] S4: Set masks for adjacent frames I1` and I2`. The mask matrix of I1` is A, the mask of the filling part is set to 0, the mask that appears in I1` but not in I2` is 1, and the rest that does not appear is 0. The mask matrix of I2` is B, and the mask of the filling part is also set to 0, the mask that appears in I2` but not in I1` is 1, and the rest that does not appear is 0.

[0047] The generation of the mask A and the mask B is to input the adjacent frames I1` and I2` into the Attention U-Net network to extract features, and then input the features into the sub-network for generating adaptive parameters, so that the mask A and the mask B can be generated.

[0048] The sub-network module for generating the mask is composed of a mixture of a convolutional layer, a ReLU function, an average downsampling layer and a Sigmoid function.

[0049] I 合成 =A*C f (I′1)+B*C b (I′2) (2)

[0050] S5: Generate an intermediate frame. The generation network uses different weights for different directions, different pixels, and distances from reference pixels to make use of the optical flow and pixel value of each pixel.

[0051]

[0052] Among them, W*H represents the resolution of the input frame after padding, w is the kernel weight of each reference pixel, α and β are the offsets of the reference pixel in the x and y directions respectively. Since the displacement of each pixel is not necessarily an integer number of pixels, α and β are learned using the frame interpolation model, and d is the distance coefficient from each pixel to the reference pixel.

[0053] The loss function in the frame interpolation model calculates the loss between the synthetic frame and the groundtruth, connects the generated frame and one of the input frames in chronological order, and uses a discriminator to distinguish the generated frame from one of the input frames.

[0054] In this embodiment, the loss between the synthetic frame and the ground truth is calculated to reduce the difference between the synthetic frame generated by this patent and the true value gt, and the L1 loss function is used, as shown in formula (4):

[0055] L1=||I 合成 -I gt ||1 (4)

[0056] In this embodiment, the discriminator is used instead of the traditional mean square error, so this part also requires a loss function L adv The purpose is to make the synthesized frame look natural in the adjacent frames, rather than its real image; since video interpolation is a temporal factor, the synthesized frame needs to look natural in the adjacent frames, rather than the gt real image; therefore, the synthetic frame generated by the model and one of the input frames are connected in time order, and the discriminator C is trained to distinguish which of the two is the generated frame with the following loss, as shown in formula (5):

[0057] -L c =log(C([I n ,I 合成 ]))+log(1-C([I 合成 ,I n+1 ])) (5)

[0058] Adversarial loss function L adv for:

[0059] L adv =C([I n ,I 合成 ])log(C([I n ,I 合成 ]))+C([I 合成 ,I n+1 ])log(C([I 合成 ,I n+1 ])) (6)

[0060] Here n is the frame number, and n is less than the total number of video frames - 1.

[0061] In this embodiment, the dataset used is the Vimeo90K dataset for training and testing. We run the present invention on a Tesla V100-SXM2, 32G GPU device.

[0062] like Figure 4 As shown, it shows the effect of generating intermediate frames by the model obtained in different training cycles, such as Figure 4 As shown, Figure 4 The left image shows the visualization of the synthetic frame obtained by using only one training cycle model, and the right image shows the visualization of the synthetic frame obtained by using fifty training cycles model. Figure 4 As shown in Figure 2, when the training cycle is 50, the generated intermediate frames have a better visual feel; when only the model obtained from one training cycle is used to synthesize the intermediate frames, Figure 3 The left image in the figure shows that serious blur and distortion occur. After experimental comparison, it is found that when the number of training cycles is 50, the effect is the best.

[0063] Table 1 shows the experimental data of the U-Net structure with and without the attention mechanism in the present invention: compared with the frame interpolation model of the traditional U-Net network.

[0064] Table 1. Comparison of Attention U-Net (left) and U-Net (right)

[0065]

[0066] As can be seen from Table 1, when the data obtained by using U-Net with the hybrid attention mechanism for feature extraction is compared with that obtained by using only U-Net for feature extraction, U-Net with the attention mechanism can perform more complete feature extraction. Under the same experimental conditions, for different categories of experimental data, the network with Attention U-Net performs better than that without Attention.

Claims

1. An adaptive frame interpolation method based on spatiotemporal attention mechanism, characterized by: The following steps are involved: S1: Input two consecutive frames I1 and I2, and perform edge filling operation on the input frames I1 and I2, wherein the edge filling operation is to fill x pixels in width and height dimensions respectively; S2: Merge the color channels of I1` and I2` after edge filling to form 6 color channels, and input them into the Attention U-Net network; Build a frame insertion model, including a U-Net network with an attention mechanism, a sub-network for generating adaptive parameters, and a generation network; The Attention U-Net generative network with a spatiotemporal attention mechanism is used to extract features from I1` and I2`. The sub-network generates adaptive parameters based on the feature estimates extracted by the Attention U-Net. The generative network synthesizes intermediate frames based on the adaptive parameters generated by the sub-network and the input frame. S3: backward convolution C of I1` b Operation plus I2` forward convolution C f Operation I 合成 , for C f and C b The operation is to use adaptive operation, as shown in formula (1): I 合成 =C f (I'1)+C b (I'2) (1) S4: Set masks for adjacent frames I1` and I2`. The mask matrix of I1` is A, the mask of the filling part is 0, and the mask that appears in I1` but not in I2` is 1, and the mask that does not appear is 0; the mask matrix of I2` is B, and the mask of the filling part is 0, and the mask that appears in I2` but not in I1` is 1, and the mask that does not appear is 0, as shown in formula (2): I 合成 =A*C f (I'1)+B*C b (I'2) (2) S5: Generate an intermediate frame, using different weights for different directions, different pixels, and distances from the reference pixel, using the optical flow and pixel value of each pixel, as shown in formula (3): Among them, w is the weight of each reference pixel, α and β are offsets. Since the displacement of each pixel is not necessarily an integer number of pixels, the frame interpolation model is used to learn α and β, and d is the distance coefficient from each pixel to the reference pixel.

2. The adaptive frame interpolation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The adaptive parameters include parameters w1, α1 and β1 of the first frame, and parameters w2, α2 and β2 of the second frame, wherein w1 and w2 represent the weight of the first frame and the weight of the second frame respectively, α1 and α2 are the offsets added by the variable convolution function in the x direction of the first frame and the second frame, β1 and β2 represent the offsets added by the variable convolution function in the y direction of the first frame and the second frame, and the adaptive parameters are all greater than 0.

3. The adaptive frame interpolation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The Attention U-Net network includes an attention mechanism, an encoder and a decoder.

4. The adaptive frame interpolation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The attention mechanism adopts a hybrid attention mechanism structure, which is integrated by a channel attention module and a spatial attention module, and jointly learns the features of the input frame from the channel and spatial dimensions respectively.

5. The adaptive frame interpolation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The hybrid Attention introduced by the encoder consists of a convolutional layer, a reLU function, and an average downsampling layer.

6. The adaptive frame interpolation method based on spatiotemporal attention mechanism according to claim 1, characterized in that: The loss function in the frame interpolation model calculates the loss between the synthetic frame and the groundtruth, connects the generated frame and one of the input frames in chronological order, and uses a discriminator to distinguish the generated frame from one of the input frames.

Images