Video frame insertion method and device, electronic equipment, storage medium and product

Abstract

The invention provides a video frame insertion method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: predicting the motion blur weight of an insertion frame between a first image frame and a second image frame based on a motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be subjected to frame insertion; generating first pixel information of an insertion frame by using the motion blur weight, the pixel information of the first image frame and the pixel information of the second image frame based on an insertion frame generation algorithm; the motion blur prediction algorithm and the frame insertion generation algorithm are trained and determined by taking the minimum loss function when the first sample image frame and the second sample image frame are subjected to sample insertion frame generation as a target. According to the scheme, when the insertion frame between the adjacent image frames is determined, the adjacent image frames are subjected to motion blur analysis, and the motion path and the motion speed can be captured by the motion blur effect, so that the insertion frame generation accuracy can be improved, the frame insertion effect is ensured, and the video fluency is improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video frame insertion method, device, electronic device, storage medium and product. Background Art

[0002] As a carrier of information transmission, video can capture dynamic scenes, but when the frame rate is set low when shooting a video, it will affect the smoothness of the final video. In the prior art, one or more frames can be added to every two frames of the original video display by motion insertion to make up for the state between the two video frames, so as to shorten the display time between each frame, thereby increasing the refresh rate of the screen and the stability of the picture, and then making the motion transition in the video more natural and improving the smoothness of the video.

[0003] However, the existing motion interpolation technology cannot meet all situations, especially when there are large differences between actions, such as fast movement speed, large angle changes or sudden changes in object behavior. The accuracy of the generated interpolation frames is low, which affects the interpolation effect and thus the smoothness of the video. Summary of the invention

[0004] Based on the above needs, the present application proposes a video interpolation method, device, electronic device, storage medium and product, which can improve the accuracy of inserted frames, ensure the interpolation effect, and thus improve the smoothness of the video.

[0005] To achieve the above objectives, this application proposes the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a video frame insertion method is provided, comprising:

[0007] Based on a motion blur prediction algorithm, predicting a motion blur weight of an interpolated frame between a first image frame and a second image frame; wherein the first image frame and the second image frame are adjacent image frames in the video to be interpolated;

[0008] Based on an interpolation frame generation algorithm, the first pixel information of the interpolation frame is generated by using the motion blur weight, the pixel information of the first image frame, and the pixel information of the second image frame;

[0009] The motion blur prediction algorithm and the interpolation frame generation algorithm are trained and determined with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame; wherein the loss function includes an optical flow estimation loss function, which is determined by utilizing the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm.

[0010] Optionally, based on a motion blur prediction algorithm, predicting a motion blur weight of an inserted frame between the first image frame and the second image frame includes:

[0011] Predicting a blur level of an interpolated frame between a first image frame and a second image frame based on a pre-trained motion blur estimation network;

[0012] Determining a motion blur weight corresponding to the blur degree of the inserted frame based on a pre-constructed mapping function;

[0013] The motion blur estimation network and the mapping function are trained and generated with the goal of minimizing the blur estimation loss function when predicting the motion blur weight of a sample inserted frame between two sample image frames; the blur estimation loss function is determined by the difference between the sample pixel information of the sample inserted frame generated by the predicted motion blur weight and the pixel information of the real intermediate frame; the predicted motion blur weight is generated by predicting the motion blur weight of the sample inserted frame between two sample image frames, and the real intermediate frame is the real image frame between the two sample image frames.

[0014] Optionally, the video frame insertion method further includes:

[0015] Collecting at least one preceding adjacent image frame of the first image frame and at least one following adjacent image frame of the second image frame from the video to be inserted, and combining the preceding adjacent image frame, the first image frame, the second image frame and the following adjacent image frame to obtain an image frame set;

[0016] Based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are fused to obtain second pixel information of the inserted frame.

[0017] Optionally, based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, including:

[0018] Determine the time weight corresponding to each image frame by using the blur degree of each image frame in the image frame set and the time difference corresponding to each image frame; wherein the time difference corresponding to the image frame is the time difference between the shooting time of the image frame and the insertion time of the insertion frame;

[0019] Based on the time weight corresponding to each image frame in the image frame set, weighted fusion is performed on each image frame in the image frame set to obtain second pixel information of the inserted frame.

[0020] Optionally, based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, including:

[0021] Based on the blur degree of the image frame in the image frame set, using a two-dimensional convolution kernel to simulate the motion blur of the image frame to obtain a two-dimensional blur kernel corresponding to the image frame;

[0022] Performing a convolution operation on an image frame in the image frame set and a two-dimensional blur kernel corresponding to the image frame to obtain a blur filter function of the image frame;

[0023] All the image frames in the image frame set are fused using the fuzzy filter function of the image frame to obtain second pixel information of the inserted frame.

[0024] Optionally, the video frame insertion method further includes:

[0025] A weighted average process is performed on the first pixel information of the inserted frame and the second pixel information of the inserted frame to obtain target pixel information of the inserted frame.

[0026] Optionally, the loss function further includes: a fuzzy degree prediction loss function;

[0027] The blur degree prediction loss function is determined by using the difference between the predicted blur degree of the sample insertion frame and the blur degree label of the real intermediate frame;

[0028] The predicted blur degree is a blur degree of a sample inserted frame between a first sample image frame and a second sample image frame predicted based on a pre-trained motion blur estimation network, and the blur degree label of the real intermediate frame is the blur degree of the real intermediate frame between the first sample image frame and the second sample image frame.

[0029] Optionally, the loss function further includes: a smoothing loss function, and the smoothing loss function includes: a spatial smoothing loss function and / or a temporal smoothing loss function;

[0030] The spatial smoothness loss function is determined based on the difference between the gradient information of the sample insertion frame and the gradient information of the real intermediate frame, and the gradient information of the sample insertion frame in the horizontal direction and the gradient information of the sample insertion frame in the vertical direction; the real intermediate frame is a real intermediate frame between the first sample image frame and the second sample image frame;

[0031] The temporal smoothness loss function is determined based on the sample insertion frame, the first sample image frame, the second sample image frame, and an insertion position parameter of the sample insertion frame between the first sample image frame and the second sample image frame.

[0032] According to a second aspect of an embodiment of the present application, a video frame insertion device is provided, including:

[0033] A blur weight prediction module, used to predict a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be inserted;

[0034] An insertion frame generation module, configured to generate first pixel information of the insertion frame based on an insertion frame generation algorithm and using the motion blur weight, the pixel information of the first image frame, and the pixel information of the second image frame;

[0035] The motion blur prediction algorithm and the interpolation frame generation algorithm are trained and determined with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame; wherein the loss function includes an optical flow estimation loss function, which is determined by utilizing the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm.

[0036] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;

[0037] The memory is connected to the processor and is used to store programs;

[0038] The processor is used to implement the above-mentioned video frame insertion method by running the program in the memory.

[0039] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned video interpolation method is implemented.

[0040] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the above-mentioned video frame insertion method.

[0041] The video interpolation method proposed in the present application predicts the motion blur weight of the interpolation frame between the first image frame and the second image frame based on the motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be interpolated; based on the interpolation generation algorithm, the first pixel information of the interpolation frame is generated by using the motion blur weight, the pixel information of the first image frame and the pixel information of the second image frame; the motion blur prediction algorithm and the interpolation generation algorithm are determined by training the minimum loss function when generating the sample interpolation frame for the first sample image frame and the second sample image frame; wherein the loss function includes the optical flow estimation loss function, which is determined by using the optical flow information of the first sample image frame, the motion blur weight label of the sample interpolation frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample interpolation frame predicted based on the motion blur prediction algorithm. By adopting the technical solution of the present application, when determining the interpolation frame between adjacent image frames, the adjacent image frames can be subjected to motion blur analysis, and the motion blur effect can capture the motion path and motion speed, thereby improving the accuracy of the interpolation frame generation, ensuring the interpolation effect, and thus improving the video fluency. Moreover, when using the loss function for algorithm training, adding motion blur to the loss function can improve the accuracy of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A schematic diagram of a flow chart of a video frame insertion method provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a processing flow for predicting motion blur weights provided in an embodiment of the present application;

[0045] Figure 3 A flowchart of another video frame insertion method provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a processing flow for determining second pixel information provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of another processing flow for determining second pixel information provided in an embodiment of the present application;

[0048] Figure 6 A flowchart of another video frame insertion method provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a video frame insertion device provided in an embodiment of the present application;

[0050] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The technical solution of the embodiment of the present application is applicable to the application scenarios of video processing and computer vision, and is specifically used in the application scenario of video frame insertion processing. The technical solution of the embodiment of the present application can improve the accuracy of inserted frames, ensure the effect of frame insertion, and thus improve the smoothness of the video.

[0052] As a carrier of information transmission, video can capture dynamic scenes. However, when the frame rate is set low, the motion difference between frames in the captured video is large. If it is in a high-dynamic scene, the motion difference between frames in the captured video will be even greater, making the final video less smooth. In addition, traditional motion detection and tracking algorithms for videos often rely on the image features of adjacent frames to remain relatively consistent. If the motion difference between adjacent image frames is large, it will affect the performance of traditional motion detection and tracking algorithms.

[0053] Therefore, in order to ensure the smoothness of the video, the prior art usually adopts the method of motion insertion to add one or more frames between two adjacent image frames displayed in the original video, so as to make up for the state between the action of two adjacent image frames, so as to shorten the display time between each frame, thereby improving the refresh rate of the screen and the stability of the picture, thereby making the action transition in the video more natural and improving the smoothness of the video.

[0054] The mainstream frame insertion algorithms at present are mainly RIFE algorithm and FILM algorithm. Among them, RIFE algorithm generates intermediate frames by estimating the optical flow between video frames, thereby improving the frame rate of the video; FILM algorithm uses a more advanced strategy of combining optical flow estimation with spatiotemporal context information, so that it can generate high-quality interpolated frames even in the case of large and complex movements. However, both RIFE algorithm and FILM algorithm cannot solve the interpolation problem when the difference between actions is large (such as too fast movement speed, large angle change or sudden change of object behavior, etc.), and both will have different degrees of artifacts, motion estimation errors or interpolation incoherence, resulting in low accuracy of the generated interpolated frames, affecting the interpolation effect, and thus affecting the smoothness of the video.

[0055] Based on this, the present application proposes a video interpolation method. This technical solution can perform motion blur analysis on adjacent image frames when determining the interpolation frame between adjacent image frames. The motion blur effect can capture the motion path and speed, thereby improving the accuracy of interpolation frame generation, ensuring the interpolation effect, and further improving the video smoothness.

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] Exemplary Methods

[0058] See also Figure 1 As shown, the embodiment of the present application proposes a video frame insertion method. The method includes:

[0059] S101. Predicting a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm.

[0060] Specifically, motion blur, as a common visual phenomenon, is usually a blur effect produced when an object moves quickly, and it can be used to represent the motion path and speed information. If the principle of motion blur can be cleverly used, the problem of excessive motion difference between two frames of video can be alleviated. Motion blur can be used to better capture the speed and direction of the object's movement, thereby improving the accuracy of optical flow estimation. Therefore, this embodiment can introduce motion blur into optical flow estimation, improve the accuracy of optical flow estimation, and further improve the accuracy of insert frame generation.

[0061] In this embodiment, the video to be interpolated is used as the video to be interpolated, and the two adjacent frames in the video to be interpolated are used as the first frame and the second frame, that is, the first frame and the second frame are adjacent frames in the video to be interpolated, and the first frame is the previous frame of the second frame. In this embodiment, a motion blur prediction algorithm is first used to predict the motion blur weight of the interpolated frame between the first frame and the second frame. Among them, the motion blur weight of the interpolated frame includes the motion blur that should be applied to each pixel in the interpolated frame. For example, for the fast-moving area in the first frame and the second frame, the interpolated frame between the two should apply a stronger motion blur, and for the static area or the slow-moving area in the first frame and the second frame, the interpolated frame between the two should apply a weaker motion blur. In this embodiment, the predicted motion blur weight includes optical flow information such as the motion speed and motion path of each pixel.

[0062] In this embodiment, the motion blur prediction algorithm can be implemented using a pre-built motion blur perception module, wherein the motion blur perception module is used to detect the motion blur area in the video image frame and dynamically adjust the blur intensity. The motion blur perception module can be constructed using a convolutional neural network (CNN) or a self-attention module. The first image frame and the second image frame are used as input frames of the motion blur perception module, and the motion blur area is detected from the input frame using the motion blur perception module, and the motion blur weight of the inserted frame between the two input frames is determined. The motion blur perception module predicts the motion blur weight based on all pixel information in the first image frame and all pixel information in the second image frame. Motion blur analysis is performed to predict the motion blur intensity of each pixel point in the inserted frame between the first image frame and the second image frame. All pixel information in the image frame includes coordinate information and pixel values ​​of each pixel point in the image frame.

[0063] In this embodiment, the motion blur prediction algorithm can adopt the following formula:

[0064] W = g(I t ,I t+1 )

[0065] Among them, I t represents the first image frame, including all pixel information in the first image frame, I t+1 represents the second image frame, including all pixel information in the second image frame, g represents the nonlinear mapping function of the motion blur perception module. W represents the predicted motion blur weight of the inserted frame, including the motion blur weight of each pixel in the inserted frame. The motion blur weight is used to characterize the motion blur intensity, that is, the larger the motion blur weight of a pixel, the stronger the motion blur of the pixel.

[0066] S102 . Based on an interpolation frame generation algorithm, first pixel information of an interpolation frame is generated by using a motion blur weight, pixel information of the first image frame, and pixel information of the second image frame.

[0067] After the motion blur weight of the inserted frame is predicted through the above steps, this embodiment needs to involve the motion blur weight of the inserted frame in the inserted frame generation process, so as to apply different motion blur effects to each pixel point of the inserted frame. Specifically, this embodiment needs to use the interpolation frame generation algorithm to generate the interpolation frame according to the motion blur weight for the pixel information of the first image frame and the pixel information of the second image frame, and obtain the first pixel information of the inserted frame. Among them, the pixel information of the first image frame includes the coordinate information and pixel value of each pixel point in the first image frame, the pixel information of the second image frame includes the coordinate information and pixel value of each pixel point in the second image frame, and the first pixel information of the inserted frame includes the coordinate information and pixel value of each pixel point in the inserted frame.

[0068] In this embodiment, the calculation formula of the interpolation frame generation algorithm is as follows:

[0069] I mid (i,j)=(1-W(i,j))I t (i,j)+W(i,j)I t+1 (i,j)

[0070] Among them, I t (i, j) represents the pixel value of the pixel at position (i, j) in the first image frame, I t+1 (i, j) represents the pixel value of the pixel at position (i, j) in the second image frame, W(i, j) represents the motion blur weight at position (i, j) in the motion blur weight of the inserted frame, and I mid (i, j) represents the pixel value of the pixel at position (i, j) in the inserted frame.

[0071] In this embodiment, the role of the motion blur weight of the inserted frame is to dynamically adjust the degree of motion blur influence of the first image frame and the second image frame according to the degree of motion of the pixel point, so as to ensure a smooth transition of the inserted frame and reflect the effect of motion blur at the same time. The motion blur weight of the inserted frame determines the degree of transition of the pixel information in the inserted frame from the first image frame to the second image frame, that is, the greater the motion blur weight of a certain pixel point, the more the pixel point in the inserted frame is biased towards the pixel point in the first image frame, and the smaller the motion blur weight of a certain pixel point, the more the pixel point in the inserted frame is biased towards the pixel point in the second image frame. During the interpolation process, the influences of the first image frame and the second image frame should be complementary, and the contributions of the two image frames need to be smoothly adjusted during the transition. In the calculation formula of the above interpolation generation algorithm, (1-W(i,j)) represents the weight of the pixel at position (i,j) in the first image frame, and W(i,j) represents the weight of the pixel at position (i,j) in the second image frame. The weighted method of the calculation formula of the interpolation generation algorithm means that the inserted frame will gradually transition from the first image frame to the second image frame as the motion blur weight changes, which helps to smooth the transition and avoid abrupt boundaries or artifacts. When the insertion frame is generated, the objects in the fast-moving area usually have a shorter exposure time and a strong blur effect. Therefore, the pixels in the fast-moving area should be more biased towards the second image frame, and a blur effect needs to be applied. In order to avoid over-sharpening, W(i,j) is used to reduce the contribution of the first image frame to the fast-moving area. The weight represented by W(i,j) is larger, and the weight represented by (1-W(i,j)) is smaller. Therefore, W(i,j) is used as the weight value of the second image frame, and (1-W(i,j)) is used as the weight value of the first image frame; on the contrary, the pixels in the relatively static area or the slow-moving area are more biased towards the first image frame to maintain the stability of the image frame. At this time, the weight represented by (1-W(i,j)) is larger, and the weight represented by W(i,j) is smaller, thereby ensuring that the insertion frame is more consistent with the first image frame. Therefore, (1-W(i,j)) is used as the weight value of the first image frame, and W(i,j) is used as the weight value of the second image frame.

[0072] Furthermore, in this embodiment, in order to ensure the application accuracy of the motion blur prediction algorithm and the interpolation frame generation algorithm, it is necessary to pre-train the motion blur prediction algorithm and the interpolation frame generation algorithm. First, it is necessary to collect the first sample image frame and the second sample image frame in the sample video, wherein the first sample image frame is located before the second sample image frame in the sample video, and the image frame between the first sample image frame and the second sample image frame is the real intermediate frame between the first sample image frame and the second sample image frame. This embodiment pre-constructs a loss function for training the motion blur prediction algorithm and the interpolation frame generation algorithm, and then trains the motion blur prediction algorithm and the interpolation frame generation algorithm with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame.

[0073] In this embodiment, the loss function pre-constructed for training the motion blur prediction algorithm and the interpolation frame generation algorithm includes an optical flow estimation loss function, which is determined by using the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted by the motion blur prediction algorithm. Among them, the sample motion blur weight of the sample insertion frame is the motion blur weight of the sample insertion frame between the first sample image frame and the second sample image frame predicted by the motion blur prediction algorithm. The motion blur weight label of the sample insertion frame is calculated by using the first sample image frame and the second sample image frame according to the motion blur calculation method.

[0074] In this embodiment, the calculation formula of the motion blur calculation method is as follows:

[0075]

[0076] Among them, B(x,y,t) represents the motion blur effect of the pixel at position (x,y) at time t, that is, the motion blur weight; T represents the exposure time, which can be obtained from the shooting parameters of the video using video processing software (such as Adobe Premiere, Final Cut Pro, etc.). x Indicates the speed of the pixel at position (x,y) in the x direction, v y Indicates the speed of the pixel at the position (x, y) in the y direction, which can be calculated according to the pixel information in the first sample image frame and the pixel information in the second sample image frame, and the interval length between the first sample image frame and the second sample image frame.

[0077] In this embodiment, the optical flow estimation loss function is as follows:

[0078]

[0079] Among them, Lflow I represents the optical flow estimation loss function; t+1 (i, j) represents the pixel value of the pixel at position (i, j) in the second sample image frame; I t represents the first sample image frame; F x (i, j) represents the optical flow information of the pixel at position (i, j) in the first sample image frame in the x direction, F y (i, j) represents the optical flow information of the pixel at position (i, j) in the first sample image frame in the y direction, wherein the optical flow information includes information such as motion trajectory and motion speed; λ is the regularization coefficient; It is used to control the smoothness of the optical flow. α is a hyperparameter used to adjust the influence of the blur guide term ‖B(i,j)-β(i,j)‖; B(i,j) represents the motion blur weight of the pixel at position (i,j) in the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method; β(i,j) represents the motion blur weight of the pixel at position (i,j) in the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm. The motion blur guide term ‖B(i,j)-β(i,j)‖ is added to the optical flow estimation loss function of this embodiment to make the optical flow estimation more consistent with the actual motion blur trajectory.

[0080] From the above introduction, it can be seen that the video interpolation method proposed in the embodiment of the present application predicts the motion blur weight of the interpolation frame between the first image frame and the second image frame based on the motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be interpolated; based on the interpolation generation algorithm, the first pixel information of the interpolation frame is generated by using the motion blur weight, the pixel information of the first image frame and the pixel information of the second image frame; the motion blur prediction algorithm and the interpolation generation algorithm are determined by training the minimum loss function when generating the sample interpolation frame for the first sample image frame and the second sample image frame; wherein the loss function includes the optical flow estimation loss function, which is determined by using the optical flow information of the first sample image frame, the motion blur weight label of the sample interpolation frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample interpolation frame predicted based on the motion blur prediction algorithm. By adopting the technical solution of the present application, when determining the interpolation frame between adjacent image frames, the adjacent image frames can be subjected to motion blur analysis, and the motion blur effect can capture the motion path and motion speed, thereby improving the accuracy of the interpolation frame generation, ensuring the interpolation effect, and thus improving the video fluency. Moreover, when using the loss function for algorithm training, adding motion blur to the loss function can improve the accuracy of the algorithm.

[0081] As an optional implementation, see Figure 2As shown, in another embodiment of the present application, step S101 is disclosed, based on a motion blur prediction algorithm, predicting a motion blur weight of an inserted frame between a first image frame and a second image frame, specifically comprising the following steps:

[0082] S201. Predicting the blur degree of an inserted frame between a first image frame and a second image frame based on a pre-trained motion blur estimation network.

[0083] In this embodiment, a motion blur estimation network is pre-built and trained. The motion blur estimation network can predict the blur degree of each pixel in the image frame inserted between the two image frames based on the two input image frames. The blur degree is a quantification of the blur property of the image and an indicator reflecting the intensity of motion blur. The blur degree can be used as a guiding parameter to affect the calculation of the motion blur weight. The blur degree and the motion blur weight complement each other. The blur degree provides an indication of the blur intensity for the interpolation process, and the motion blur weight can be determined according to the blur degree. In this embodiment, the motion blur estimation network preferably uses a deep convolutional network.

[0084] Specifically, in this embodiment, both the first image frame and the second image frame are input into a motion blur estimation network, and the motion blur estimation network predicts the blur degree of each pixel based on the motion change between the first image frame and the second image frame as the blur degree of the inserted frame between the first image frame and the second image frame. The specific prediction process of the motion blur estimation network includes: first, using the convolution layer in the motion blur estimation network to extract features of the first image frame and the second image frame respectively, including extracting basic information such as edges and textures of the images, using the activation function (such as RELU) in the motion blur estimation network to capture the nonlinear features of the first image frame and the second image frame, and using the pooling layer in the motion blur estimation network to perform spatial dimensionality reduction on the extracted features to capture higher-level motion information of the first image frame and the second image frame. Then, the motion blur estimation network estimates the motion between the first image frame and the second image frame based on the extracted feature information of the first image frame and the second image frame, especially the pixel-level motion change, that is, the motion of each pixel between the first image frame and the second image frame is estimated by the optical flow estimation algorithm. The optical flow field can describe the speed and direction of each pixel. The motion blur estimation network determines the blur degree based on the optical flow field, thereby predicting the blur degree of each pixel in the interpolated frame between the first image frame and the second image frame. The last layer in the motion blur estimation network is a regression layer, which is used to output the blur degree of each pixel in the interpolated frame between the first image frame and the second image frame. This output can be a scalar value, which is often normalized to [0,1], where 0 represents no blur and 1 represents maximum blur.

[0085] In this embodiment, the calculation formula of the motion blur estimation network is as follows:

[0086] M=f(I t ,I t+1 )

[0087] Wherein, M represents the blur degree of the inserted frame between the first image frame and the second image frame; I t represents a first image frame, including pixel information of each pixel point in the first image frame; t+1 represents the second image frame, which includes pixel information of each pixel in the second image frame; f is a motion blur estimation network, which is used to predict the blur degree of each pixel in the inserted frame.

[0088] S202: Determine a motion blur weight corresponding to the blur degree of the inserted frame based on a pre-constructed mapping function.

[0089] In this embodiment, a mapping function is pre-constructed, and the mapping function stores the conversion rule of blur degree into motion blur weight. The mapping function preferably adopts a linear mapping function, that is, the value of blur degree is linearly mapped to motion blur weight, and a larger blur degree corresponds to a larger motion blur weight.

[0090] In order to improve the accuracy of blur degree prediction and motion blur weight prediction, the present embodiment pre-sets a blur estimation loss function for training the motion blur estimation network and the mapping function. For the training of the motion blur estimation network and the mapping function, the goal is to minimize the blur estimation loss function when predicting the motion blur weight of the sample inserted frame between two sample image frames. Specifically, it is necessary to first collect a sample video, extract two sample image frames from the sample video, and use the image frame between the two sample image frames as the real intermediate frame between the two sample image frames. Then, both sample image frames are input into the motion blur estimation network, and the blur degree of the sample inserted frame between the two sample image frames is predicted. The motion blur weight corresponding to the blur degree of the sample inserted frame is determined by using the mapping function. Then, based on the interpolation frame generation algorithm, the motion blur weight of the sample inserted frame and the pixel information of the two sample image frames are used to generate the sample pixel information of the sample inserted frame. Finally, the motion blur estimation network and the mapping function are trained with the goal of minimizing the difference between the sample pixel information of the sample inserted frame and the pixel information of the real intermediate frame. In the present embodiment, the formula of the blur estimation loss function is as follows:

[0091]

[0092] Where L(W) represents the fuzzy estimation loss function; I ncw (x, y) represents the pixel information at the position (x, y) in the sample insertion frame;groundtruth (x,y) represents the pixel information at position (x,y) in the real intermediate frame.

[0093] As an optional implementation, the present application embodiment also proposes a video frame insertion method, see Figure 3 As shown, the method includes:

[0094] S301 . Predicting a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm.

[0095] S302 : Based on an interpolation frame generation algorithm, first pixel information of an interpolation frame is generated by using a motion blur weight, pixel information of the first image frame, and pixel information of the second image frame.

[0096] S303, collecting at least one preceding adjacent image frame of the first image frame and at least one succeeding adjacent image frame of the second image frame from the video to be inserted, and combining the preceding adjacent image frame, the first image frame, the second image frame and the succeeding adjacent image frame to obtain an image frame set.

[0097] When determining the interpolation frame between the first image frame and the second image frame, this embodiment can combine the context information of the first image frame and the second image frame, and by analyzing the image frames at multiple time points, it can extract richer motion and scene features, so that the generated interpolation frame is more natural and accurate. Therefore, after obtaining the first image frame and the second image frame that need to be interpolated from the video to be interpolated, this embodiment also needs to collect at least one front adjacent image frame of the first image frame and at least one rear adjacent image frame of the second image frame from the video to be interpolated, and then combine the front adjacent image frame, the first image frame, the second image frame and the rear adjacent image frame to obtain an image frame set. For example, the image frame set can be {I t-2 ,I t-1 ,I t ,I t+1 ,I t+2 ,I t+3}, where I t is the first image frame, I t+1 is the second image frame, I t-2 ,I t-1 are all previous adjacent image frames, I t+2 ,I t+3 They are all subsequent adjacent image frames.

[0098] S304: Based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain second pixel information of the inserted frame.

[0099] This embodiment needs to predict the blur degree of each image frame in the image frame set, determine the blur degree of each image frame in the image frame set, and then fuse all the image frames in the image frame set based on the blur degree of each image frame to obtain the second pixel information of the inserted frame, that is, according to the blur degree of each image frame, fuse the various motion blurs in the time dimension, realize the combination of context information, and thus generate the inserted frame between the first image frame and the second image frame. This embodiment uses multiple image frames to perform motion blur fusion in the time dimension, so as to enhance the coherence of the inserted frame and reduce the jitter and distortion of the inserted frame.

[0100] In this embodiment, the blur degree of each image frame in the image frame set can be predicted by inputting the previous image frame and the next image frame of the image frame into a pre-trained motion blur estimation network to predict the blur degree of the image frame. In addition, if the image frame is the first image frame or the last image frame in the video to be inserted, then the image frame does not have a next image frame or a previous image frame. At this time, the pre-set boundary frame blur degree is directly used as the blur degree of the image frame. Among them, the motion blur estimation network used in this embodiment has been specifically described in the above embodiment, and will not be repeated in this embodiment.

[0101] In this embodiment, the insertion frame with the first pixel information generated in step S302 can be used as the final insertion frame between the first image frame and the second image frame, and the insertion frame with the second pixel information generated in step S304 can be used as the final insertion frame between the first image frame and the second image frame.

[0102] The specific execution mode of step S301 and step S302 in this embodiment is the same as the specific execution mode of step S101 and step S102 in the above embodiment, and will not be described in detail in this embodiment. This embodiment does not limit the execution order of steps S301-S302 and steps S303-S304, that is, steps S301-S302 may be executed first, and then steps S303-S304; steps S303-S304 may be executed first, and then steps S301-S302; steps S301-S302 and steps S303-S304 may also be executed simultaneously.

[0103] As an optional implementation, see Figure 4 As shown, in another embodiment of the present application, step S304 is disclosed, based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, which specifically includes the following steps:

[0104] S401 : Determine the time weight corresponding to each image frame by using the blur degree of each image frame in the image frame set and the time difference corresponding to each image frame.

[0105] This embodiment needs to predict the blur degree of each image frame in the image frame set, and also needs to calculate the time difference corresponding to each image frame in the image frame set. Among them, the method of predicting the blur degree of the image frame has been specifically described in the above embodiment, and this embodiment will not be repeated. The time difference corresponding to the image frame is the time difference between the shooting time of the image frame and the insertion time of the inserted frame. Since the time difference corresponding to the image frame and the blur degree of the image frame will affect the proportion of the image frame during fusion, therefore, this embodiment needs to use the blur degree of each image frame in the image frame set and the time difference corresponding to each image frame to determine the time weight corresponding to each image frame, wherein the time weight corresponding to the image frame represents the contribution of the image frame to the inserted frame.

[0106] In this embodiment, the calculation formula of the time weight corresponding to the image frame is as follows:

[0107]

[0108] Where W(i,j) represents the time weight of the pixel at position (i,j) in the image frame; M(i,j) represents the blur degree of the pixel at position (I,J) in the image frame; t represents the shooting time of the image frame, t mid Indicates the insertion time of the inserted frame, |tt mid | represents the time difference between the shooting time of the image frame and the insertion time of the inserted frame; α represents the constant that adjusts the influence of the blur degree on the time weight. The calculation formula of the time weight shows that the longer the shooting time of the image frame is from the insertion time of the inserted frame, the smaller the time weight is, and the inserted frame should rely more on the image frame whose shooting time is closer to its insertion time; the greater the motion blur of the image frame, the smaller the time weight is, and the generation of the inserted frame relies more on the blur effect of the subsequent frame; the smaller the motion blur of the image frame, the larger the time weight is, and the inserted frame relies more on the blur effect of the previous frame.

[0109] S402: Perform weighted fusion on each image frame in the image frame set based on the time weight corresponding to each image frame in the image frame set to obtain second pixel information of the inserted frame.

[0110] After determining the time weights corresponding to each image frame in the image frame set through the above steps, the time weight is used as the fusion weight of each image frame, and all the image frames in the image frame set are weightedly fused, so as to obtain the second pixel information of the inserted frame, so that the generation of the inserted frame can combine the context information of the first image frame and the second image frame, thereby improving the generation accuracy of the inserted frame and the smooth transition of the inserted frame.

[0111] In this embodiment, the calculation formula for weighted fusion of each image frame in the image frame set is as follows:

[0112]

[0113] Among them, I mid represents the inserted frame between the first image frame and the second image frame, including the second pixel information of the inserted frame; M represents the sequence number of the first image frame in the image frame set, and N represents the sequence number of the last image frame in the image frame set; W k represents the time weight corresponding to the k-th image frame; I k represents the k-th image frame, including pixel information of the k-th image frame.

[0114] As an optional implementation, see Figure 5 As shown, in another embodiment of the present application, step S304 is disclosed, based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, which may specifically include the following steps:

[0115] S501. Based on the blur degree of an image frame in an image frame set, a two-dimensional convolution kernel is used to simulate motion blur of the image frame to obtain a two-dimensional blur kernel corresponding to the image frame.

[0116] After predicting the blur degree of each image frame in the image frame set, this embodiment needs to determine the two-dimensional blur kernel corresponding to each image frame in the image frame set. Among them, determining the two-dimensional blur kernel corresponding to the image frame requires using a two-dimensional convolution kernel to simulate the motion blur of the image frame. The two-dimensional blur kernel can be generated by rotation or translation to simulate the real motion trajectory of the object. The formula is as follows:

[0117]

[0118] Where H(x,y) represents the two-dimensional blur kernel of the pixel at position (x,y) in the image frame; d x Indicates the horizontal displacement of the pixel at position (x, y) in the image frame, d y represents the vertical displacement of the pixel at position (x, y) in the image frame; L represents the blur level of the image frame. x It is the product of the horizontal velocity of the pixel point at the position in the image frame and the time interval corresponding to the image frame, d yIt is the product of the vertical component velocity of the pixel point at the position in the image frame and the time interval corresponding to the image frame, where the time interval corresponding to the image frame refers to the time interval between two image frames used to calculate the blur degree of the image frame.

[0119] S502 : performing a convolution operation on image frames in the image frame set and two-dimensional blur kernels corresponding to the image frames to obtain blur filter functions of the image frames.

[0120] After defining the two-dimensional blur kernel corresponding to the image frame through the above steps, the two-dimensional blur kernel corresponding to the image frame is convolved with the image frame to achieve a blur effect, thereby obtaining a blur filter function of the image frame. The convolution operation can be expressed as:

[0121]

[0122] Where (x, y) represents the pixel at position (x, y) in the image frame, (i, j) represents the position in the two-dimensional blur kernel, H(i, j) represents the weight of position (i, j) in the two-dimensional blur kernel, and F blur (x,y) represents the blur filter function of the image frame.

[0123] S503: Using the fuzzy filter function of the image frame, all the image frames in the image frame set are merged to obtain second pixel information of the inserted frame.

[0124] After the fuzzy filter function of each image frame in the image frame set is determined through the above steps, all the image frames in the image frame set are fused using the fuzzy filter function of the image frame to obtain the second pixel information of the inserted frame. Wherein, all the image frames in the image frame set are fused, that is, the fuzzy filter functions of all the image frames in the image frame set are averaged.

[0125] In this embodiment, the calculation formula for fusing all the image frames in the image frame set is as follows:

[0126]

[0127] Among them, I mid represents an interpolated frame between the first image frame and the second image frame, including pixel information of the interpolated frame; K represents the number of image frames included in the image frame set; I k represents the k-th image frame in the image frame set, including the pixel information of the k-th image frame; v k represents the motion speed of the kth image frame in the image frame set, including the motion speed of each pixel point of the kth image frame, and the motion speed includes the component speed in the horizontal direction and the component speed in the vertical direction; F represents the blur filter function.

[0128] As an optional implementation, the embodiment of the present application also proposes a video frame insertion method.

[0129] See also Figure 6 As shown, the method includes:

[0130] S601 . Predicting a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm.

[0131] S602: Based on an interpolation frame generation algorithm, first pixel information of an interpolation frame is generated by using a motion blur weight, pixel information of the first image frame, and pixel information of the second image frame.

[0132] S603: Collect at least one preceding adjacent image frame of the first image frame and at least one succeeding adjacent image frame of the second image frame from the video to be inserted, and combine the preceding adjacent image frame, the first image frame, the second image frame and the succeeding adjacent image frame to obtain an image frame set.

[0133] S604: Based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are merged to obtain second pixel information of the inserted frame.

[0134] S605 , performing weighted average processing on the first pixel information of the inserted frame and the second pixel information of the inserted frame to obtain target pixel information of the inserted frame.

[0135] After calculating the first pixel information and the second pixel information of the inserted frame, this embodiment performs weighted averaging processing on the first pixel information of the inserted frame and the second pixel information of the inserted frame to obtain target pixel information of the inserted frame, wherein the weight of the first pixel information and the weight of the second pixel information are determined according to the accuracy of the two. For example, the first pixel information is only calculated based on the first image frame and the second image frame, while the second pixel information combines the context information of the first image frame and the second image frame. Therefore, the accuracy of the second pixel information may be higher than the accuracy of the first pixel information. Therefore, the weight of the second pixel information is preferably greater than the weight of the first pixel information.

[0136] In addition, if the calculation of the second pixel information uses Figure 4 The calculation method described above was used to calculate, and also used Figure 5 The calculation method described above is used to calculate. Then, when determining the target pixel information of the inserted frame, it is necessary to calculate the first pixel information of the inserted frame and the target pixel information of the inserted frame. Figure 4 The second pixel information of the inserted frame calculated by the calculation method and the Figure 5 The second pixel information of the inserted frame calculated by the calculation method is subjected to weighted average processing, and the calculation formula is as follows:

[0137] Ifinal =w 1 I mid,1 +w 2 I mid,2 +w 3 I mid,3

[0138] Among them, I final Indicates the target pixel information of the inserted frame; mid,1 Indicates the first pixel information of the inserted frame, w 1 I represents the weight corresponding to the first pixel information; mid,2 Indicates the use of Figure 4 The second pixel information of the inserted frame calculated by the calculation method, w 2 Indicates the use of Figure 4 The weight corresponding to the second pixel information of the inserted frame calculated by the calculation method; mid,3 Indicates the use of Figure 5 The second pixel information of the inserted frame calculated by the calculation method, w 3 Indicates the use of Figure 5 The weight corresponding to the second pixel information of the inserted frame calculated by the calculation method.

[0139] The specific implementation of steps S601 to S604 in this embodiment is the same as the specific implementation of steps S301 to S304 in the above embodiment, and will not be repeated in this embodiment.

[0140] As an optional implementation, another embodiment of the present application discloses that according to Figure 2 and Figure 3 It can be seen that in the process of generating the inserted frame, a blur degree prediction step is performed. Therefore, in this embodiment, the loss function used to train the motion blur prediction algorithm and the inserted frame generation algorithm also includes: a blur degree prediction loss function.

[0141] The blur prediction loss function is determined by using the difference between the predicted blur of the sample inserted frame and the blur label of the real intermediate frame. The predicted blur is the blur of the sample inserted frame between the first sample image frame and the second sample image frame predicted based on the pre-trained motion blur estimation network, and the blur label of the real intermediate frame is the blur of the real intermediate frame between the first sample image frame and the second sample image frame.

[0142] Specifically, the blur degree prediction loss function is:

[0143]

[0144] Among them, L motion_blurrepresents the blur degree prediction loss function; N represents the total number of pixels contained in the image frame; M pred (i) represents the blur level of the i-th pixel in the predicted blur level of the sample inserted into the frame; M true (i) represents the blur degree of the i-th pixel in the blur degree label of the real intermediate frame.

[0145] As an optional implementation, another embodiment of the present application discloses that in order to ensure that the generation of inserted frames remains smooth and natural in both space and time, reduce "faults" or mutations in the interpolation process, and make the transition of inserted frames smoother, it is necessary to set a smoothing loss function when training the motion blur prediction algorithm and the interpolation generation algorithm. That is, in this embodiment, the loss function used to train the motion blur prediction algorithm and the interpolation generation algorithm also includes: a smoothing loss function, and the smoothing loss function includes a spatial smoothing loss function and / or a temporal smoothing loss function.

[0146] Spatial smoothness loss is mainly used to reduce the mutation between adjacent pixels in the inserted frame, so that the brightness or color changes in the inserted frame are more natural. Spatial smoothness loss includes two aspects. One is the gradient loss, which is determined based on the difference between the gradient information of the sample inserted frame and the gradient information of the real intermediate frame. The gradient loss calculates the gradient of the sample inserted frame and compares it with the gradient of the real intermediate frame to ensure that the inserted frame has a similar structure in space when it is generated; the other is the total variation loss, which is determined based on the gradient information of the sample inserted frame in the horizontal direction and the gradient information of the sample inserted frame in the vertical direction. The total variation loss makes the image smooth and reduces the mutation area of ​​the image frame by reducing the high-frequency noise in the image frame.

[0147] Specifically, the gradient loss in the spatial smoothness loss function is:

[0148]

[0149] Among them, L grad Represents the gradient loss in the spatial smoothness loss function; I pred represents a sample insertion frame between the first sample image frame and the second sample image frame; I true represents a true intermediate frame between the first sample image frame and the second sample image frame; Represents a gradient operator, such as the Sobel operator or the Laplace operator. The Sobel operator is used to calculate the horizontal and vertical gradients of each pixel in the image frame. The Laplace operator is used to calculate the second-order derivative of the image frame to capture the edge information in the image frame.

[0150] The total variation loss in the spatial smoothness loss function is:

[0151]

[0152] Among them, L TV Represents the total variation loss in the spatial smoothness loss function; represents the gradient of the sample insertion frame between the first sample image frame and the second sample image frame in the horizontal direction, Represents the gradient of the sample insertion frame between the first sample image frame and the second sample image frame in the vertical direction. The goal of this total variation loss is to minimize the non-smooth area of ​​the insertion frame in space to make the image smoother.

[0153] The temporal smoothness loss function is to ensure that the visual content of the inserted frame is continuous with the changes of the previous and next frames by defining a temporal difference loss. The temporal smoothness loss function is determined based on the sample inserted frame, the first sample image frame, the second sample image frame, and the insertion position parameters of the sample inserted frame between the first sample image frame and the second sample image frame. The temporal smoothness loss function is:

[0154]

[0155] Among them, L time represents the time smoothing loss function; I mid represents a sample insertion frame between the first sample image frame and the second sample image frame; I t represents the first sample image frame, I t+1 represents the second sample image frame; α represents the insertion position parameter of the sample insertion frame between the first sample image frame and the second sample image frame. For example, if the insertion position of the sample insertion frame between the first sample image frame and the second sample image frame is an intermediate position, α is determined to be 0.5.

[0156] In this embodiment, the spatial smoothness loss function and the temporal smoothness loss function are used in combination to ensure that the generation of the inserted frame is not only smooth in space but also continuous in time. Therefore, the calculation formula of the smoothness loss function is as follows:

[0157] L smooth =λ 1 L TV +λ 2 L grad +λ 3 L time

[0158] Among them, L smooth represents the smooth loss function; L grad represents the gradient loss in the spatial smoothness loss function, λ 2 represents the weight of the gradient loss; L TV represents the total variation loss in the spatial smoothness loss function, λ 1 represents the weight of the total variation loss; Ltime represents the time smoothing loss function, λ 3 Represents the weight of the temporal smoothing loss function.

[0159] In this embodiment, the loss function used to train the motion blur prediction algorithm and the interpolation frame generation algorithm may include: an optical flow estimation loss function, a blur degree prediction loss function, and a smoothing loss function. The calculation formula is as follows:

[0160] L=L flow +αL motion_blur +βL smooth

[0161] Among them, L flow represents the optical flow estimation loss function; L motion_blur represents the blur degree prediction loss function, which is used to control the rationality of the blur effect; L smooth represents the smooth loss function, which is used to control the smooth transition of the inserted frame; α and β are the weight coefficients of the loss function.

[0162] Exemplary Devices

[0163] Accordingly, the embodiment of the present application also provides a video frame insertion device, see Figure 7 As shown, the device comprises:

[0164] The blur weight prediction module 100 is used to predict the motion blur weight of the inserted frame between the first image frame and the second image frame based on the motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be inserted;

[0165] The insertion frame generation module 110 is used to generate first pixel information of the insertion frame based on the insertion frame generation algorithm by using the motion blur weight, the pixel information of the first image frame and the pixel information of the second image frame;

[0166] The motion blur prediction algorithm and the interpolation frame generation algorithm are trained and determined with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame; wherein the loss function includes an optical flow estimation loss function, which is determined by utilizing the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm.

[0167] From the above introduction, it can be seen that the video interpolation device proposed in the embodiment of the present application can perform motion blur analysis on adjacent image frames when determining the interpolation frame between adjacent image frames. The motion blur effect can capture the motion path and motion speed, thereby improving the accuracy of interpolation frame generation, ensuring the interpolation effect, and further improving the video smoothness. Moreover, when using the loss function for algorithm training, adding motion blur to the loss function can improve the accuracy of the algorithm.

[0168] As an optional implementation, another embodiment of the present application discloses that the fuzzy weight prediction module 100 includes: a fuzzy degree prediction unit and a weight mapping unit.

[0169] A blur degree prediction unit, configured to predict a blur degree of an inserted frame between a first image frame and a second image frame based on a pre-trained motion blur estimation network;

[0170] A weight mapping unit, for determining a motion blur weight corresponding to a blur degree of an inserted frame based on a pre-built mapping function;

[0171] The motion blur estimation network and mapping function are trained and generated with the goal of minimizing the blur estimation loss function when predicting the motion blur weight of a sample inserted frame between two sample image frames; the blur estimation loss function is determined by the difference between the sample pixel information of the sample inserted frame generated by the predicted motion blur weight and the pixel information of the real intermediate frame; the predicted motion blur weight is generated by predicting the motion blur weight of the sample inserted frame between two sample image frames, and the real intermediate frame is the real image frame between the two sample image frames.

[0172] As an optional implementation, another embodiment of the present application discloses that the video frame insertion device further includes: an image frame acquisition module and a fusion module.

[0173] An image frame acquisition module is used to acquire at least one preceding adjacent image frame of a first image frame and at least one following adjacent image frame of a second image frame from the video to be interpolated, and to combine the preceding adjacent image frame, the first image frame, the second image frame and the following adjacent image frame to obtain an image frame set;

[0174] The fusion module is used to fuse all the image frames in the image frame set based on the blur degree of each image frame in the image frame set to obtain the second pixel information of the inserted frame.

[0175] As an optional implementation, another embodiment of the present application discloses that the fusion module includes: a time weight determination unit and a first fusion unit.

[0176] A time weight determination unit, used to determine the time weight corresponding to each image frame by using the blur degree of each image frame in the image frame set and the time difference corresponding to each image frame; wherein the time difference corresponding to the image frame is the time difference between the shooting time of the image frame and the insertion time of the insertion frame;

[0177] The first fusion unit is used to perform weighted fusion on each image frame in the image frame set based on the time weight corresponding to each image frame in the image frame set to obtain second pixel information of the inserted frame.

[0178] As an optional implementation, it is disclosed in another embodiment of the present application that the fusion module further includes: a motion blur simulation unit, a convolution operation unit and a second fusion unit.

[0179] A motion blur simulation unit, used for simulating the motion blur of the image frame using a two-dimensional convolution kernel based on the blur degree of the image frame in the image frame set, to obtain a two-dimensional blur kernel corresponding to the image frame;

[0180] A convolution operation unit, used for performing a convolution operation on an image frame in the image frame set and a two-dimensional blur kernel corresponding to the image frame to obtain a blur filter function of the image frame;

[0181] The second fusion unit is used to fuse all the image frames in the image frame set by using the fuzzy filter function of the image frame to obtain second pixel information of the inserted frame.

[0182] As an optional implementation, another embodiment of the present application discloses that the video frame insertion device further includes: a processing module.

[0183] The processing module is used to perform weighted average processing on the first pixel information of the inserted frame and the second pixel information of the inserted frame to obtain the target pixel information of the inserted frame.

[0184] As an optional implementation, in another embodiment of the present application, it is disclosed that the loss function further includes: a blur degree prediction loss function;

[0185] The blur prediction loss function is determined by using the difference between the predicted blur of the sample inserted frame and the blur label of the real intermediate frame;

[0186] The predicted blur degree is based on a pre-trained motion blur estimation network, which predicts the blur degree of a sample inserted frame between a first sample image frame and a second sample image frame. The blur degree label of the real intermediate frame is the blur degree of the real intermediate frame between the first sample image frame and the second sample image frame.

[0187] As an optional implementation, another embodiment of the present application discloses that the loss function further includes: a smoothing loss function, and the smoothing loss function includes: a spatial smoothing loss function and / or a temporal smoothing loss function;

[0188] The spatial smoothness loss function is determined based on the difference between the gradient information of the sample insertion frame and the gradient information of the real intermediate frame, as well as the gradient information of the sample insertion frame in the horizontal direction and the gradient information of the sample insertion frame in the vertical direction; the real intermediate frame is a real intermediate frame between the first sample image frame and the second sample image frame;

[0189] The temporal smoothing loss function is determined based on the sample insertion frame, the first sample image frame, the second sample image frame, and an insertion position parameter of the sample insertion frame between the first sample image frame and the second sample image frame.

[0190] The video frame insertion device provided in this embodiment belongs to the same application concept as the video frame insertion method provided in the above embodiments of this application, and can execute the video frame insertion method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the video frame insertion method. For technical details not fully described in this embodiment, please refer to the specific processing content of the video frame insertion method provided in the above embodiments of this application, and will not be repeated here.

[0191] Exemplary Electronic Devices

[0192] Another embodiment of the present application also provides an electronic device, see Figure 8 As shown, the device includes:

[0193] Memory 200 and processor 210;

[0194] The memory 200 is connected to the processor 210 and is used to store programs;

[0195] The processor 210 is used to implement the video frame insertion method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0196] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0197] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0198] A bus may include a pathway that transfers information between components of a computer system.

[0199] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0200] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0201] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0202] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0203] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0204] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0205] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any video frame insertion method provided in the above embodiments of the present application.

[0206] Exemplary computer program products and storage media

[0207] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the video interpolation method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0208] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0209] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the video interpolation method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0210] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0211] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0212] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0213] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0214] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0215] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0216] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0217] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0218] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0219] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0220] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video frame insertion method, characterized in that: include: Based on a motion blur prediction algorithm, predicting a motion blur weight of an interpolated frame between a first image frame and a second image frame; wherein the first image frame and the second image frame are adjacent image frames in the video to be interpolated; Based on an interpolation frame generation algorithm, the first pixel information of the interpolation frame is generated by using the motion blur weight, the pixel information of the first image frame, and the pixel information of the second image frame; The motion blur prediction algorithm and the interpolation frame generation algorithm are trained and determined with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame; wherein the loss function includes an optical flow estimation loss function, which is determined by utilizing the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm.

2. The method according to claim 1, characterized in that: Predicting a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm includes: Predicting a blur level of an interpolated frame between a first image frame and a second image frame based on a pre-trained motion blur estimation network; Determining a motion blur weight corresponding to the blur degree of the inserted frame based on a pre-constructed mapping function; The motion blur estimation network and the mapping function are trained and generated with the goal of minimizing the blur estimation loss function when predicting the motion blur weight of a sample inserted frame between two sample image frames; the blur estimation loss function is determined by the difference between the sample pixel information of the sample inserted frame generated by the predicted motion blur weight and the pixel information of the real intermediate frame; the predicted motion blur weight is generated by predicting the motion blur weight of the sample inserted frame between two sample image frames, and the real intermediate frame is the real image frame between the two sample image frames.

3. The method according to claim 1, characterized in that Also includes: Collecting at least one preceding adjacent image frame of the first image frame and at least one following adjacent image frame of the second image frame from the video to be inserted, and combining the preceding adjacent image frame, the first image frame, the second image frame and the following adjacent image frame to obtain an image frame set; Based on the blur degree of each image frame in the image frame set, all image frames in the image frame set are fused to obtain second pixel information of the inserted frame.

4. The method according to claim 3, characterized in that Based on the blur degree of each image frame in the image frame set, all the image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, including: Determine the time weight corresponding to each image frame by using the blur degree of each image frame in the image frame set and the time difference corresponding to each image frame; wherein the time difference corresponding to the image frame is the time difference between the shooting time of the image frame and the insertion time of the insertion frame; Based on the time weight corresponding to each image frame in the image frame set, weighted fusion is performed on each image frame in the image frame set to obtain second pixel information of the inserted frame.

5. The method according to claim 3, characterized in that: Based on the blur degree of each image frame in the image frame set, all the image frames in the image frame set are merged to obtain the second pixel information of the inserted frame, including: Based on the blur degree of the image frame in the image frame set, using a two-dimensional convolution kernel to simulate the motion blur of the image frame to obtain a two-dimensional blur kernel corresponding to the image frame; Performing a convolution operation on an image frame in the image frame set and a two-dimensional blur kernel corresponding to the image frame to obtain a blur filter function of the image frame; All the image frames in the image frame set are fused using the fuzzy filter function of the image frame to obtain second pixel information of the inserted frame.

6. The method according to claim 3, characterized in that: Also includes: A weighted average process is performed on the first pixel information of the inserted frame and the second pixel information of the inserted frame to obtain target pixel information of the inserted frame.

7. The method according to claim 2, characterized in that: The loss function also includes: a fuzzy degree prediction loss function; The blur degree prediction loss function is determined by using the difference between the predicted blur degree of the sample insertion frame and the blur degree label of the real intermediate frame; The predicted blur degree is a blur degree of a sample inserted frame between a first sample image frame and a second sample image frame predicted based on a pre-trained motion blur estimation network, and the blur degree label of the real intermediate frame is the blur degree of the real intermediate frame between the first sample image frame and the second sample image frame.

8. The method according to claim 1, characterized in that The loss function further includes: a smooth loss function, and the smooth loss function includes: a spatial smooth loss function and / or a temporal smooth loss function; The spatial smoothness loss function is determined based on the difference between the gradient information of the sample insertion frame and the gradient information of the real intermediate frame, and the gradient information of the sample insertion frame in the horizontal direction and the gradient information of the sample insertion frame in the vertical direction; the real intermediate frame is a real intermediate frame between the first sample image frame and the second sample image frame; The temporal smoothness loss function is determined based on the sample insertion frame, the first sample image frame, the second sample image frame, and an insertion position parameter of the sample insertion frame between the first sample image frame and the second sample image frame.

9. A video frame insertion device, characterized in that: include: A blur weight prediction module, used to predict a motion blur weight of an inserted frame between a first image frame and a second image frame based on a motion blur prediction algorithm; wherein the first image frame and the second image frame are adjacent image frames in the video to be inserted; An insertion frame generation module, configured to generate first pixel information of the insertion frame based on an insertion frame generation algorithm and using the motion blur weight, the pixel information of the first image frame, and the pixel information of the second image frame; The motion blur prediction algorithm and the interpolation frame generation algorithm are trained and determined with the goal of minimizing the loss function when generating sample insertion frames for the first sample image frame and the second sample image frame; wherein the loss function includes an optical flow estimation loss function, which is determined by utilizing the optical flow information of the first sample image frame, the motion blur weight label of the sample insertion frame calculated based on the motion blur calculation method, and the sample motion blur weight of the sample insertion frame predicted based on the motion blur prediction algorithm.

10. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the video frame insertion method as described in any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the video frame insertion method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the video frame insertion method as claimed in any one of claims 1 to 8.

Images