Model training method, video frame insertion method and device

Abstract

The embodiment of the invention provides a model training method, a video frame insertion method and a video frame insertion device. The method comprises the following steps: determining training data comprising a first frame, a second frame and a third frame; inputting training data into a neural network model comprising a frame insertion network and an optical flow determination network, wherein the training process comprises the following steps: the frame insertion network generates a first optical flow from a first frame to an intermediate state estimation frame and a second optical flow from a third frame to the intermediate state estimation frame; determining a network output from the optical streams: a third optical stream from the first frame to the second frame, a fourth optical stream from the third frame to the second frame, and a fifth optical stream from the intermediate state estimation frame to the second frame; determining a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow; determining a second estimated optical flow from the third frame to the second frame based on the second optical flow and the fifth optical flow; a loss function value is determined based on the first estimated optical flow and the second estimated optical flow to complete training. The problem that the time domain and the space domain are not matched in video frame insertion is solved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular to a model training method, a video frame insertion method and a device. Background Art

[0002] Video interpolation technology is to add one or more frames to every two frames of the original video to shorten the display time between each frame, correct the illusion of visual persistence of the human eye, and thus improve the smoothness of the video. After extracting frames in the form of n (n is greater than or equal to 1) frames for high frame rate video, a low frame rate video can be obtained. For the convenience of description, it is assumed that n is equal to 1, and the three adjacent frames in the high frame rate video are defined as Ft0, Ft1 and Ft2. Therefore, the interpolation processing of low frame rate video is to predict the Ft1 frame based on the Ft0 frame and the Ft2 frame.

[0003] The current video interpolation technology is mainly based on deep learning, using the true value of the Ft1 frame in the high frame rate video as a label to guide the prediction of the Ft1 frame based on the Ft0 frame and the Ft2 frame. However, this processing method has a common ill-posed problem: there may be multiple intermediate frames with different spatial states between the Ft0 frame and the Ft2 frame of similar scenes. Directly using the true value of the Ft1 frame in a specific high frame rate video as a label for training will lead to ambiguity in network learning. Summary of the invention

[0004] The embodiments of the present invention provide a model training method, a video frame insertion method, and a device.

[0005] The technical solution of the embodiment of the present invention is as follows:

[0006] A model training method, comprising:

[0007] Determine training data, the training data comprising a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame;

[0008] The training data is input into a neural network model including a frame insertion network and an optical flow determination network, and a training process of the neural network model is performed, wherein the training process includes:

[0009] Inputting the first frame and the third frame into the interpolation network, the interpolation network generating a first optical flow from the first frame to an intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein a sum of the first optical flow and the second optical flow is zero, and outputting the intermediate state estimation frame from the interpolation network;

[0010] Inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting from the optical flow determination network: a third optical flow from the first frame to the second frame, a fourth optical flow from the third frame to the second frame, and a fifth optical flow from the intermediate state estimation frame to the second frame;

[0011] Determine a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow;

[0012] Determining a second estimated optical flow from the third frame to the second frame based on the second optical flow and the fifth optical flow;

[0013] Determining a loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow;

[0014] Parameters of the interpolation network and the optical flow determination network are configured so that the loss function value is lower than a predetermined threshold.

[0015] In one embodiment, determining a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow includes:

[0016] Based on the fifth optical flow, reverse sampling is performed on the first optical flow to obtain a first reverse sampling result; the fifth optical flow is added to the first reverse sampling result to obtain the first estimated optical flow;

[0017] The determining, based on the second optical flow and the fifth optical flow, a second estimated optical flow from the third frame to the second frame comprises:

[0018] Based on the fifth optical flow, reverse sampling is performed on the second optical flow to obtain a second reverse sampling result; and the fifth optical flow is added to the second reverse sampling result to obtain the second estimated optical flow.

[0019] In one embodiment, outputting an intermediate state estimation frame between the first frame and the third frame from the interpolation network includes:

[0020] Based on the first optical flow, reverse sampling is performed on the feature map of the first frame to obtain a third reverse sampling result;

[0021] Based on the second optical flow, reverse sampling is performed on the feature map of the third frame to obtain a fourth reverse sampling result;

[0022] The intermediate state estimation frame is determined based on the third inverse sampling result and the fourth inverse sampling result.

[0023] In one embodiment, determining the loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow includes:

[0024] determining a third difference between a first estimated frame and the second frame, wherein the first estimated frame is determined based on the first estimated optical flow and the first frame;

[0025] determining a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame;

[0026] determining a fifth difference between the first optical flow and the inverse of the second optical flow;

[0027] The loss function value is determined based on the first difference, the second difference, the third difference, the fourth difference and the fifth difference.

[0028] A video frame insertion method, comprising:

[0029] Extracting a trained interpolation network from a neural network model trained according to any of the above model training methods;

[0030] Determining a first test frame and a second test frame based on a time sequence;

[0031] Inputting the first test frame and the second test frame into the trained interpolation network, and receiving an intermediate state test estimation frame output from the trained interpolation network;

[0032] The intermediate state test estimation frame is inserted between the first test frame and the second test frame.

[0033] A model training device, comprising:

[0034] A determination module, configured to determine training data, wherein the training data includes a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame;

[0035] A training module is used to input the training data into a neural network model including a frame insertion network and an optical flow determination network, and execute a training process of the neural network model, wherein the training process includes:

[0036] Inputting the first frame and the third frame into the interpolation network, the interpolation network generating a first optical flow from the first frame to an intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein a sum of the first optical flow and the second optical flow is zero, and outputting the intermediate state estimation frame from the interpolation network;

[0037] Inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting from the optical flow determination network: a third optical flow from the first frame to the second frame, a fourth optical flow from the third frame to the second frame, and a fifth optical flow from the intermediate state estimation frame to the second frame;

[0038] Determine a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow;

[0039] Determining a second estimated optical flow from the third frame to the second frame based on the second optical flow and the fifth optical flow;

[0040] Determining a loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow;

[0041] Parameters of the interpolation network and the optical flow determination network are configured so that the loss function value is lower than a predetermined threshold.

[0042] In one embodiment, the training module is used to perform reverse sampling on the first optical flow based on the fifth optical flow to obtain a first reverse sampling result; add the fifth optical flow to the first reverse sampling result to obtain the first estimated optical flow; perform reverse sampling on the second optical flow based on the fifth optical flow to obtain a second reverse sampling result; add the fifth optical flow to the second reverse sampling result to obtain the second estimated optical flow.

[0043] In one embodiment, the training module is used to perform reverse sampling on the feature map of the first frame based on the first optical flow to obtain a third reverse sampling result; perform reverse sampling on the feature map of the third frame based on the second optical flow to obtain a fourth reverse sampling result; and determine the intermediate state estimation frame based on the third reverse sampling result and the fourth reverse sampling result.

[0044] In one embodiment, the training module is used to determine a third difference between a first estimated frame and the second frame, wherein the first estimated frame is determined based on a first estimated optical flow and the first frame; determine a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame; determine a fifth difference between the first optical flow and the opposite number of the second optical flow; and determine the loss function value based on the first difference, the second difference, the third difference, the fourth difference and the fifth difference.

[0045] A video frame insertion device, comprising:

[0046] An extraction module, used to extract a trained interpolation network from a neural network model trained according to any of the above model training methods;

[0047] A determination module, used to determine a first test frame and a second test frame based on a time sequence;

[0048] An input module, configured to input the first test frame and the second test frame into the trained interpolation network, and receive an intermediate state test estimation frame output from the trained interpolation network;

[0049] An inserting module is used to insert the intermediate state test estimation frame between the first test frame and the second test frame.

[0050] It can be seen from the above technical scheme that in the implementation mode of the present invention, different from the prior art that directly uses the intermediate frames of the high frame rate video as labels, the implementation mode of the present invention outputs an intermediate state estimation frame (the first optical flow and the second optical flow whose sum result is zero are fused), and the intermediate state estimation frame and the intermediate frame of the high frame rate video are jointly input into the optical flow determination network, and a loss function is constructed by constructing a first estimated optical flow from the first frame to the second frame (determined based on the first optical flow and the fifth optical flow) and a second estimated optical flow from the third frame to the second frame (determined based on the second optical flow and the fifth optical flow), so that the intermediate state estimation frame is more accurate. Therefore, the implementation mode of the present invention can generate an interpolated frame of the intermediate state, solves the problem of mismatch between the time domain and the spatial domain in the video interpolated frame, and improves the network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Schematic diagram of a training interpolation network in the prior art.

[0052] Figure 2 It is an exemplary flow chart of the model training method according to an embodiment of the present invention.

[0053] Figure 3 It is a schematic diagram of the training process of the interpolation network according to an embodiment of the present invention.

[0054] Figure 4 It is an exemplary schematic diagram of the video frame insertion process according to an embodiment of the present invention.

[0055] Figure 5 It is an exemplary structural diagram of a model training device according to an embodiment of the present invention.

[0056] Figure 6 It is an exemplary structural diagram of a video frame insertion device according to an embodiment of the present invention.

[0057] Figure 7 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0059] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0060] In the current video interpolation technology, the middle frame between the two reference frames in the high frame rate video is directly used as a label to guide the prediction process of the middle frame based on the two reference frames. However, this processing method leads to ambiguity in network learning.

[0061] Figure 1 It is a schematic diagram of a training interpolation network in the prior art. Assume that the three adjacent frames in a high frame rate video are Ft0 frame, Ft1 frame and Ft2 frame. The Ft0 frame and the Ft2 frame are input into the interpolation network to be trained as a video frame pair (Ft0, Ft2). The convolutional network in the interpolation network generates an optical flow (Flow0m) from the Ft0 frame to the predicted frame and an optical flow (Flow2m) from the Ft2 frame to the predicted frame based on the Ft0 frame and the Ft2 frame. Then, the interpolation network fuses Flow0m and Flow2m to obtain a predicted frame Ftm. Next, the loss function of the interpolation network is determined based on the difference between the predicted frame Ftm and the Ft1 frame as the label, and the loss function is used to perform back propagation to update the parameters of the interpolation network so that the output of the interpolation network is close to the Ft1 frame, thereby completing the training.

[0062] However, there may be multiple Ft1 frames of different states between (Ft0, Ft2) with similar scenes. This training method that uses the true value of the Ft1 frame in a specific high frame rate video as a label will cause the network learning to be ambiguous. Specifically: the default Ft0 frame, Ft1 frame and Ft2 frame satisfy 2*t1=t0+t2 in the time domain, where t1 is the timestamp of the Ft1 frame, t0 is the timestamp of the Ft0 frame, and t2 is the timestamp of the Ft2 frame. However, for a moving object in a specific high frame rate video, it does not strictly guarantee that the spatial position of the moving object satisfies 2*s(t1)=s(t0)+s(t2), where s(·) represents displacement. Therefore, from the perspective of the high frame rate video itself, it does not necessarily satisfy the consistency of the time domain and the spatial domain. Therefore, for video interpolation in similar scenes, if the Ft1 frame in the high frame rate video is used as the true value to directly guide the adjacent Ft0 and Ft2 of the low frame rate video to restore Ft1, it is ambiguous.

[0063] For example: Assume that in a high frame rate video, a fast-moving football moves from point A to point B, the Ft0 frame records the position of the ball at point A, the Ft2 frame records the position of the ball at point B, and the Ft1 frame records the position of the football at the intermediate time point ((t0+t2) / 2)) between A and B. However, during the period of t0-t2, as the football moves from A to B, affected by different initial velocities and accelerations, at the intermediate time point, the football may actually be at any spatial position between points A and B, and not necessarily at the position recorded by the Ft1 frame in the high frame rate video. Therefore, there is ambiguity in using the Ft1 frame in the high frame rate video as a label to complete training, which is also a pathological problem that has not been solved in the current field of video interpolation.

[0064] The above disclosure details the technical defects existing in the prior art, the causes of the technical defects and the thinking and analysis process of overcoming the technical defects. In fact, the cognition of the above technical defects is not common knowledge in the field, but a novel discovery made by the applicant in research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the applicant in the actual research process, and are not common knowledge in the field.

[0065] Figure 2 is an exemplary flow chart of a model training method according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0066] Step 101: Determine training data, where the training data includes a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame.

[0067] Step 102: Input the training data into the neural network model including the interpolation network and the optical flow determination network, and execute the training process of the neural network model. Specifically, the training process includes: inputting the first frame and the third frame into the interpolation network, the interpolation network generates a first optical flow from the first frame to the intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein the sum of the first optical flow and the second optical flow is zero, and outputting the intermediate state estimation frame from the interpolation network; inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting from the optical flow determination network: the third optical flow from the first frame to the second frame, the fourth optical flow from the third frame to the second frame, and the fifth optical flow from the intermediate state estimation frame to the second frame; based on the first optical flow and the fifth optical flow, determining the first estimated optical flow from the first frame to the second frame; based on the second optical flow and the fifth optical flow, determining the second estimated optical flow from the third frame to the second frame; based on the first difference between the first estimated optical flow and the third optical flow and the second difference between the second estimated optical flow and the fourth optical flow, determining the loss function value; configuring the parameters of the interpolation network and the optical flow determination network so that the loss function value is lower than a predetermined threshold.

[0068] It can be seen that, unlike the prior art that directly uses the intermediate frames of the high frame rate video as labels, the implementation method of the present invention outputs an intermediate state estimation frame (the first optical flow and the second optical flow whose sum result is zero are fused), and the intermediate state estimation frame and the intermediate frame of the high frame rate video are jointly input into the optical flow determination network, and the loss function is jointly constructed by the first estimated optical flow from the first frame to the second frame (determined based on the first optical flow and the fifth optical flow) and the second estimated optical flow from the third frame to the second frame (determined based on the second optical flow and the fifth optical flow), so that the intermediate state estimation frame is more accurate. Therefore, the implementation method of the present invention can generate accurate intermediate state interpolation frames, can solve the problem of mismatch between the time domain and the spatial domain in video interpolation, and improve network performance.

[0069] In one embodiment, based on the first optical flow and the fifth optical flow, determining the first estimated optical flow from the first frame to the second frame includes: based on the fifth optical flow, performing backward sampling on the first optical flow to obtain a first backward sampling result; adding the fifth optical flow to the first backward sampling result to obtain the first estimated optical flow; based on the second optical flow and the fifth optical flow, determining the second estimated optical flow from the third frame to the second frame includes: based on the fifth optical flow, performing backward sampling on the second optical flow to obtain a second backward sampling result; adding the fifth optical flow to the second backward sampling result to obtain the second estimated optical flow.

[0070] Therefore, obtaining the first estimated optical flow from the first frame to the second frame and the second estimated optical flow from the third frame to the second frame by reverse sampling has the advantage of simple implementation.

[0071] In one embodiment, outputting an intermediate state estimation frame between a first frame and a third frame from an interpolation network includes: performing reverse sampling on a feature map of the first frame based on a first optical flow to obtain a third reverse sampling result; performing reverse sampling on a feature map of the third frame based on a second optical flow to obtain a fourth reverse sampling result; and determining an intermediate state estimation frame based on the third reverse sampling result and the fourth reverse sampling result.

[0072] Therefore, through the reverse sampling method, an estimated frame based on the first optical flow (characterized by the third reverse sampling result) and an estimated frame based on the second optical flow (characterized by the fourth reverse sampling result) can be obtained, and then the final intermediate state estimation frame is obtained based on these two estimated frames. This not only has the advantage of simple implementation, but also utilizes the bidirectional optical flow to estimate the intermediate frame, thereby improving the accuracy of the estimation.

[0073] In one embodiment, determining a loss function value based on a first difference between a first estimated optical flow and a third optical flow and a second difference between a second estimated optical flow and a fourth optical flow includes: determining a third difference between a first estimated frame and a second frame, wherein the first estimated frame is determined based on the first estimated optical flow and the first frame; determining a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame; determining a fifth difference between the opposite numbers of the first optical flow and the second optical flow; and determining a loss function value based on the first difference, the second difference, the third difference, the fourth difference, and the fifth difference.

[0074] It can be seen that in the loss function design of the implementation mode of the present invention, not only the L1 loss between the first estimated optical flow and the third optical flow, and between the second estimated optical flow and the fourth optical flow (represented by the first difference and the second difference) is taken into account, but also the reconstruction loss of the interpolated frame (represented by the third difference and the fourth difference) and the L1 loss of the intermediate state estimation frame (represented by the fifth difference) are considered. Therefore, the loss consideration is more comprehensive, and the accuracy of the network is improved.

[0075] The training process and the testing process of the implementation mode of the present invention are exemplarily described below.

[0076] First, let the optical flow from Fti frame to Ftj frame be Flowij, where the image coordinate system of Flowij is consistent with the image coordinate system of Ftj frame. The estimated frame Ftj' of Ftj frame realized by Fti frame and Flowij is:

[0077] Ftj'= Backward Sampling(Fti, Flowij) Formula (1)

[0078] Among them: the dimensions of Fti, Ftj and Ftj' are 1*C*H*W; the dimension of flowij is 1*2*H*W; C, H and W are data channels, height and width respectively, and 1 represents the batch size (Batchsize).

[0079] Moreover, the absolute value error L1 between the Ftj' frame and the Ftj frame can be expressed as:

[0080] L1 = |Ftj- Ftj'| Formula (2)

[0081] Let the intermediate time between time ti and time tj be tm; the corresponding intermediate state frame in the spatial domain is Ftm; the optical flow from Fti to Ftm is Flowim; the optical flow from Ftj to Ftm is Fjm; where Flowim and Fowjm are opposite numbers.

[0082] Figure 3 It is a schematic diagram of the training process of the interpolation network according to an embodiment of the present invention.

[0083] Assume that the three adjacent frames in the high frame rate video are Ft0, Ft1 and Ft2, where Ft1 is in the middle of Ft0 and Ft2, Ft0 is before Ft1, and Ft2 is after Ft1. The neural network model includes a frame insertion network and an optical flow determination network.

[0084] Specifically, the training process includes:

[0085] Step 1: Take Ft0 and Ft2 as a video frame pair (Ft0, Ft2) and input them into the interpolation network to be trained. The convolutional network in the interpolation network generates the optical flow (Flow0m) from Ft0 to the intermediate state prediction frame and the optical flow (Flow2m) from Ft2 to the intermediate state prediction frame based on Ft0 and Ft2. Then, in the interpolation network, Flow0m and Flow2m are fused to obtain the intermediate state prediction frame Ftm.

[0086] For example: Ftm=0.5*(Backward Sampling(Ft0,Flow0m)+Backward Sampling(Ft2,Flow2m)), and Flow0m+Flow2m=0; wherein Backward Sampling(.) is the reverse sampling function.

[0087] Step 2: Form a video frame pair (Ft0, Ft1) based on the Ft0 frame and the Ft1 frame, form a video frame pair (Ft2, Ft1) based on the Ft2 frame and the Ft1 frame, and form a video frame pair (Ftm, Ft1) based on the Ftm frame and the Ft1 frame, and input these three video frame pairs (Ft0, Ft1), (Ft2, Ft1) and (Ftm, Ft1) into the optical flow determination network, and obtain the optical flow Flow01 from the Ft0 frame to the Ft1 frame, the optical flow Flow21 from the Ft2 frame to the Ft1 frame, and the optical flow Flowm1 from the Ftm frame to the Ft1 frame respectively.

[0088] Step 3: Calculate the estimated optical flow Flow01' from the Ft0 frame to the Ft1 frame and the estimated optical flow Flow21' from the Ft2 frame to the Ft1 frame. Where:

[0089] Flow01'=Flowm1+Backward Sampling(Flow0m,Flowm1);

[0090] Flow21'=Flowm1+Backward Sampling(Flow2m,Flowm1).

[0091] Step 4: Based on formula (1) and formula (2), calculate the first difference D1 between the estimated optical flow Flow01' and Flow01, the second difference D2 between the estimated optical flow Flow21' and Flow21, the third difference D3 between the estimated frame determined based on Flow01' and Ft0 and Ft1, the fourth difference D4 between the estimated frame determined based on Flow21' and Ft2 and Ft1, and the fifth difference D5 between Flow0m and -Flow2m. Where:

[0092] D1 = |Flow01'-Flow01|;

[0093] D2 = |Flow21'-Flow21|;

[0094] D3=|Backward Sampling(Ft0,Flow01')-Flow01|;

[0095] D4=|Backward Sampling(Ft2,Flow21')-Flow01|;

[0096] D5=|Flow0m-(-Flow2m)|=|Flow0m+Flow2m|.

[0097] Step 5: Determine the loss function value Loss based on the first difference D1, the second difference D2, the third difference D3, the fourth difference D4 and the fifth difference D5.

[0098] Where: Loss = a1*D1+a2*D2+a3*D3+a4*D4+a5*D5. Where: a1, a2, a3, a4 and a5 are preset weights respectively. Preferably, the values ​​of a1, a2, a3, a4 and a5 are all adjustable.

[0099] Step 6: Perform back propagation using the loss function value Loss to update the parameters in the interpolation network and the optical flow determination network, wherein the training is completed when the loss function value Loss is less than a predetermined threshold.

[0100] After completing the above training process, the interpolation network can be used to perform specific video interpolation tasks.

[0101] The embodiment of the present invention further proposes a video frame insertion method. The video frame insertion method comprises:

[0102] Step 1: Extract the trained interpolation network from the neural network model trained according to the model training method described above.

[0103] Step 2: Determine the first test frame and the second test frame based on the time sequence.

[0104] Step 3: Input the first test frame and the second test frame into the trained interpolation network, and receive the intermediate state test estimation frame output from the trained interpolation network.

[0105] Step 4: Insert the intermediate state test estimation frame between the first test frame and the second test frame.

[0106] Figure 4 It is an exemplary schematic diagram of the video frame insertion process according to an embodiment of the present invention.

[0107] exist Figure 4 In the example, based on the Ft0 frame and the Ft2 frame as test data, a video frame pair (Ft0, Ft2) is formed, and the video frame pair (Ft0, Ft2) is input to perform the following Figure 3 The trained interpolation network after the training process is shown. The convolutional network in the trained interpolation network generates the optical flow (Flow0m) from the Ft0 frame to the intermediate state prediction frame and the optical flow (Flow2m) from the Ft2 frame to the intermediate state prediction frame based on the Ft0 frame and the Ft2 frame. Then, in the trained interpolation network, Flow0m and Flow2m are fused to obtain the intermediate state prediction frame Ftm. Then, the intermediate state prediction frame Ftm can be inserted between the Ft0 frame and the Ft2 frame, thereby realizing video interpolation and ensuring the matching of the time domain and the spatial domain in the video interpolation.

[0108] Figure 5 is an exemplary structural diagram of a model training device according to an embodiment of the present invention. Figure 5 As shown, the model training device 500 includes:

[0109] A determination module 501 is used to determine training data, where the training data includes a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame;

[0110] The training module 502 is used to input the training data into the neural network model including the interpolation network and the optical flow determination network, and execute the training process of the neural network model, wherein the training process includes: inputting the first frame and the third frame into the interpolation network, the interpolation network generates a first optical flow from the first frame to the intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein the sum of the first optical flow and the second optical flow is zero, and the intermediate state estimation frame is output from the interpolation network; inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting the intermediate state estimation frame from the optical flow determination network Output: a third optical flow from the first frame to the second frame, a fourth optical flow from the third frame to the second frame, and a fifth optical flow from the intermediate state estimated frame to the second frame; based on the first optical flow and the fifth optical flow, determine the first estimated optical flow from the first frame to the second frame; based on the second optical flow and the fifth optical flow, determine the second estimated optical flow from the third frame to the second frame; based on the first difference between the first estimated optical flow and the third optical flow and the second difference between the second estimated optical flow and the fourth optical flow, determine the loss function value; configure the parameters of the interpolation network and the optical flow determination network so that the loss function value is lower than a predetermined threshold.

[0111] In one embodiment, the training module 502 is used to perform reverse sampling on the first optical flow based on the fifth optical flow to obtain a first reverse sampling result; add the fifth optical flow to the first reverse sampling result to obtain a first estimated optical flow; perform reverse sampling on the second optical flow based on the fifth optical flow to obtain a second reverse sampling result; add the fifth optical flow to the second reverse sampling result to obtain a second estimated optical flow.

[0112] In one embodiment, the training module 502 is used to perform reverse sampling on the feature map of the first frame based on the first optical flow to obtain a third reverse sampling result; perform reverse sampling on the feature map of the third frame based on the second optical flow to obtain a fourth reverse sampling result; and determine an intermediate state estimation frame based on the third reverse sampling result and the fourth reverse sampling result.

[0113] In one embodiment, the training module 502 is used to determine a third difference between a first estimated frame and a second frame, wherein the first estimated frame is determined based on a first estimated optical flow and the first frame; determine a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame; determine a fifth difference between the first optical flow and the opposite number of the second optical flow; and determine a loss function value based on the first difference, the second difference, the third difference, the fourth difference and the fifth difference.

[0114] Figure 6 FIG. 1 is an exemplary structural diagram of a video frame insertion device according to an embodiment of the present invention. Figure 6As shown, the video interpolation device 600 includes: an extraction module 601, used to extract a trained interpolation network from a neural network model trained according to any of the above model training methods; a determination module 602, used to determine a first test frame and a second test frame based on a time sequence; an input module 603, used to input the first test frame and the second test frame into the trained interpolation network, and receive an intermediate state test estimation frame output from the trained interpolation network; an insertion module 604, used to insert the intermediate state test estimation frame between the first test frame and the second test frame.

[0115] Figure 7 It is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. The electronic device 700 includes: a processor 701 and a memory 702. The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 701 may also include a main processor and a coprocessor, the main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI processor, which is used to process computing operations related to machine learning. For example, the AI ​​processor can be implemented as a neural network processor.

[0116] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices.

[0117] In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 701 to implement the model training method or video interpolation method provided in each embodiment of the present disclosure. In some embodiments, the electronic device 700 may also optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a touch display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709. The peripheral device interface 703 can be used to connect at least one peripheral device related to input / output (I / O) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0118] The radio frequency circuit 704 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a wireless fidelity (Wireless Fidelity, Wi-Fi) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to near field communication (NFC), which is not limited in the present disclosure.

[0119] The display screen 705 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, set on the front panel of the electronic device 700; in other embodiments, the display screen 705 can be at least two, respectively set on different surfaces of the electronic device 700 or in a folding design; in some embodiments, the display screen 705 can be a flexible display screen, set on the curved surface or folding surface of the electronic device 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as a liquid crystal display (LCD) and an organic light-emitting diode (OLED).

[0120] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and the virtual reality (VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash may be a monochrome temperature flash or a dual color temperature flash. A dual color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0121] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into a sound wave audible to humans, but also convert the electrical signal into a sound wave inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack. The positioning component 708 is used to locate the current geographical location of the electronic device 700 to achieve navigation or location-based services (LBS). Positioning component 708 may be a positioning component based on the Global Positioning System (GPS) of the United States, the Beidou system of China, the Grenas system of Russia, or the Galileo system of the European Union. Power supply 709 is used to power various components in electronic device 700. Power supply 709 may be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 709 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging.

[0122] Those skilled in the art will appreciate that the above structure does not constitute a limitation on the electronic device 700, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements. It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and certain steps or modules may be ignored according to actual needs. The execution order of each step is not fixed and may be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module may be implemented by multiple modules, and the functions of multiple modules may also be implemented by the same module. These modules may be located in the same device or in different devices. The hardware modules in each embodiment may be implemented mechanically or electronically. For example, a hardware module may include a specially designed permanent circuit or logic device (such as a dedicated processor, such as an FPGA or ASIC) for completing a specific operation. The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software for performing a specific operation. As for the specific mechanical method, or the use of a dedicated permanent circuit, or the use of a temporarily configured circuit (such as configured by software) to implement the hardware module, it can be determined based on cost and time considerations.

[0123] The present invention also provides a machine-readable storage medium, storing instructions for making a machine execute such as the method of the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for realizing the function of any of the above-mentioned embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system etc. operated on the computer can also be completed part or all of the actual operation by instructions based on the program code. The program code read out from the storage medium can also be written to the memory provided in the expansion board inserted into the computer or to the memory provided in the expansion unit connected to the computer, and then the CPU etc. installed on the expansion board or the expansion unit are made to perform part and all of the actual operation based on the instructions of the program code, so as to realize the function of any of the above-mentioned embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0124] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: include: Determine training data, the training data comprising a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame; The training data is input into a neural network model including a frame insertion network and an optical flow determination network, and a training process of the neural network model is performed, wherein the training process includes: Inputting the first frame and the third frame into the interpolation network, the interpolation network generating a first optical flow from the first frame to an intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein a sum of the first optical flow and the second optical flow is zero, and outputting the intermediate state estimation frame from the interpolation network; Inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting from the optical flow determination network: a third optical flow from the first frame to the second frame, a fourth optical flow from the third frame to the second frame, and a fifth optical flow from the intermediate state estimation frame to the second frame; Determine a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow; Determining a second estimated optical flow from the third frame to the second frame based on the second optical flow and the fifth optical flow; Determining a loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow; Parameters of the interpolation network and the optical flow determination network are configured so that the loss function value is lower than a predetermined threshold.

2. The method according to claim 1, characterized in that The determining, based on the first optical flow and the fifth optical flow, a first estimated optical flow from the first frame to the second frame comprises: Based on the fifth optical flow, reverse sampling is performed on the first optical flow to obtain a first reverse sampling result; the fifth optical flow is added to the first reverse sampling result to obtain the first estimated optical flow; The determining, based on the second optical flow and the fifth optical flow, a second estimated optical flow from the third frame to the second frame comprises: Based on the fifth optical flow, reverse sampling is performed on the second optical flow to obtain a second reverse sampling result; and the fifth optical flow is added to the second reverse sampling result to obtain the second estimated optical flow.

3. The method according to claim 1, characterized in that Outputting an intermediate state estimation frame between the first frame and the third frame from the interpolation network includes: Based on the first optical flow, reverse sampling is performed on the feature map of the first frame to obtain a third reverse sampling result; Based on the second optical flow, reverse sampling is performed on the feature map of the third frame to obtain a fourth reverse sampling result; The intermediate state estimation frame is determined based on the third inverse sampling result and the fourth inverse sampling result.

4. The method according to claim 1, characterized in that: The determining of the loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow comprises: determining a third difference between a first estimated frame and the second frame, wherein the first estimated frame is determined based on the first estimated optical flow and the first frame; determining a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame; determining a fifth difference between the first optical flow and the inverse of the second optical flow; The loss function value is determined based on the first difference, the second difference, the third difference, the fourth difference and the fifth difference.

5. A video frame insertion method, characterized in that: include: Extracting a trained interpolation network from a neural network model trained by the model training method according to any one of claims 1 to 4; Determine a first test frame and a second test frame based on a time sequence; Inputting the first test frame and the second test frame into the trained interpolation network, and receiving an intermediate state test estimation frame output from the trained interpolation network; The intermediate state test estimation frame is inserted between the first test frame and the second test frame.

6. A model training device, characterized in that: include: A determination module, configured to determine training data, wherein the training data includes a first frame, a second frame, and a third frame based on a time sequence, wherein the second frame is between the first frame and the third frame; A training module is used to input the training data into a neural network model including a frame insertion network and an optical flow determination network, and execute a training process of the neural network model, wherein the training process includes: Inputting the first frame and the third frame into the interpolation network, the interpolation network generating a first optical flow from the first frame to an intermediate state estimation frame and a second optical flow from the third frame to the intermediate state estimation frame, wherein a sum of the first optical flow and the second optical flow is zero, and outputting the intermediate state estimation frame from the interpolation network; Inputting the first frame, the second frame, the third frame and the intermediate state estimation frame into the optical flow determination network, and outputting from the optical flow determination network: a third optical flow from the first frame to the second frame, a fourth optical flow from the third frame to the second frame, and a fifth optical flow from the intermediate state estimation frame to the second frame; Determine a first estimated optical flow from the first frame to the second frame based on the first optical flow and the fifth optical flow; Determining a second estimated optical flow from the third frame to the second frame based on the second optical flow and the fifth optical flow; Determining a loss function value based on a first difference between the first estimated optical flow and the third optical flow and a second difference between the second estimated optical flow and the fourth optical flow; Parameters of the interpolation network and the optical flow determination network are configured so that the loss function value is lower than a predetermined threshold.

7. The device according to claim 6, characterized in that The training module is configured to perform reverse sampling on the first optical flow based on the fifth optical flow to obtain a first reverse sampling result; and add the fifth optical flow to the first reverse sampling result to obtain the first estimated optical flow; Based on the fifth optical flow, performing reverse sampling on the second optical flow to obtain a second reverse sampling result; The fifth optical flow is added to the second reverse sampling result to obtain the second estimated optical flow.

8. The device according to claim 6, characterized in that The training module is used to perform reverse sampling on the feature map of the first frame based on the first optical flow to obtain a third reverse sampling result; and perform reverse sampling on the feature map of the third frame based on the second optical flow to obtain a fourth reverse sampling result; The intermediate state estimation frame is determined based on the third inverse sampling result and the fourth inverse sampling result.

9. The device according to claim 6, characterized in that The training module is used to determine a third difference between a first estimated frame and the second frame, wherein the first estimated frame is determined based on the first estimated optical flow and the first frame; determining a fourth difference between a second estimated frame and the second frame, wherein the second estimated frame is determined based on the second estimated optical flow and the third frame; determining a fifth difference between the first optical flow and the inverse of the second optical flow; The loss function value is determined based on the first difference, the second difference, the third difference, the fourth difference and the fifth difference.

10. A video frame insertion device, characterized in that: include: An extraction module, used to extract a trained interpolation network from a neural network model trained according to the model training method according to any one of claims 1 to 4; A determination module, used to determine a first test frame and a second test frame based on a time sequence; An input module, configured to input the first test frame and the second test frame into the trained interpolation network, and receive an intermediate state test estimation frame output from the trained interpolation network; An inserting module is used to insert the intermediate state test estimation frame between the first test frame and the second test frame.

Images