Unsupervised video frame insertion method based on depth information and random Dropout

Abstract

The invention discloses an unsupervised video frame insertion method based on depth information and random Dropout. The unsupervised video frame insertion method comprises the following steps: step 1, generating a prediction frame It through model; step 2, preprocessing the training set of the training model; step 3, generating a prediction frame through model; step 4, updating loss value calculation of the model; 5, updating the model through the obtained loss value; and step 6, testing the trained model by using the test data. According to the method, the Dropout layer is added to the expert model, so that the influence caused by overfitting is reduced; in addition, depth information is introduced to judge the front-back relation of the object, the fuzzy problem caused by object shielding is solved, and the prediction effect of the model is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of video processing, and in particular relates to an unsupervised video frame insertion method based on depth information and random Dropout. Background Art

[0002] In the field of video processing, in order to improve the smoothness and viewing experience of the video, it is necessary to perform interpolation processing on the original video. The current video interpolation method is mainly based on the training of neural network models. This requires supervised training models for high-frequency videos. With the development of society, the demand for high-frame rate videos in different scenes continues to increase. At the same time, obtaining a large number of high-frame rate videos requires high costs (storage space, power consumption, etc.) and devices that can obtain high-frame rate videos. How to train a video interpolation model without relevant high-frame rate video acquisition methods has become a hot issue. At present, in order to address the problem of high model training costs, a method of using a large amount of data to train a model is generally used to improve its generalization ability, but this method also causes the model to not reflect the characteristics of the current video that needs to be interpolated well. For this reason, a method of using unsupervised technology to train the model has emerged.

[0003] Video Interpolation (VI) is a method used in the field of computer graphics to increase the frame rate of videos. This method is mainly based on convolutional neural networks. It uses algorithms to automatically learn the movement and change rules between video frames to generate high-quality intermediate frames. When high-frame rate videos are not available, unsupervised video interpolation technology is required. Unsupervised video interpolation technology does not require a large number of labeled data sets for training. It only generates intermediate frames by automatically learning the inherent rules between video frames. Current video interpolation technology is mainly trained through high-frame rate video supervision. Its methods include Figure 1 As shown in the figure, first, a low frame rate (Low FPS) video is input; then, the bidirectional optical flow network is used to generate the optical flow information of two frames of images, and the intermediate optical flow information is calculated; then, the optical flow information is used to calculate the intermediate interpolated frame image to obtain a high frame rate video (High FPS); finally, the high frame rate video is used for supervision to calculate the loss value to update the model parameters, and finally a model that can perform video interpolation is obtained.

[0004] At present, many experts and scholars have studied video interpolation methods: in 2018, Jiang et al. achieved high-quality video intermediate frame estimation by training deep neural networks; in 2024, researchers from Shanghai Jiaotong University and other institutions proposed a new perception-oriented video frame interpolation paradigm that effectively solved the inaccurate motion estimation and the alignment of supervision signals. However, when it is impossible to obtain relevant high-frame rate videos for supervision, unsupervised training methods are needed, but unsupervised training methods lack some basic facts. For this reason, Reda et al. proposed using expert models for pseudo supervision (Pseudo Supervised, PS) in 2019. However, the expert model used in this method is not trained with the same type of video, and the model will have some unnecessary features. Therefore, the model trained using the pseudo-supervised method will have a certain overfitting phenomenon. In addition, since the final result of the synthesis requires a view map to determine whether there is occlusion, the method proposed by Reda et al. does not have a loss value for directly supervising the view map, which will aggravate the background blur at the edge of the object. In summary, how to reduce the influence of the expert model on the training results during unsupervised training of the interpolation model and strengthen the judgment of image occlusion problems is an important issue that affects the effect of the video interpolation model. Summary of the invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention provides an unsupervised video interpolation method based on depth information and random Dropout to solve the problem of video interpolation in an unsupervised environment. A Dropout layer is added to the expert model to reduce the impact of overfitting. The present invention also proposes to introduce depth information to judge the front and back relationship of objects, solve the blur problem caused by object occlusion, and further improve the prediction effect of the model.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] An unsupervised video frame insertion method based on depth information and random Dropout includes the following steps:

[0008] Step 1: Generate prediction frame I through model t : Extract two consecutive video frames (I0, I1) from the low frame rate video to be inserted and generate a predicted frame. The function of the model is to input two frames of images I0, I1 and output the inserted frame I0 at any time t between the two frames. t , expressed as:

[0009]

[0010] in, Indicates the model mode of training;

[0011] Step 2: Preprocess the training set of the training model, input the high frame rate training set TRAIN_DATA, the number of interpolation frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER;

[0012] Step 3: Generate a prediction frame through the model, input a set of data (I0, I1, I2) in TRAIN_LOADER, the model to be trained, and output the prediction frame

[0013] Step 4: Update the loss value calculation of the model;

[0014] Step 5. Update the model with the obtained loss value: The model is updated by gradient update, and the optimizer is Adam. After each training scene, optimizer.step() is called to update the parameters. Training all the video frames is counted as one cycle, and the sum of the training loss values ​​of each scene in one cycle is calculated. This value is used as the basis for updating the learning rate. The updating method is ReduceLROnPlateau. When the learning rate is less than 1e-7, the training ends and the model is saved.

[0015] Step 6: Use the test data to test the trained model.

[0016] Further, the process of step one is as follows:

[0017] Step 1.1: Obtain the optical flow and field of view map of the input video frame through the neural network model. In the optical flow mapping module, g(*) represents the use of optical flow F t→0 and F t→1 Map the input frame, optical flow (F t→0 ,F t→1 ) The optical flow (F) between two frames generated by U-net1 0→1 ,F 1→0 ) to make an approximate estimate;

[0018] Assuming that the change of optical flow is smooth, the solid optical flow line F between video frames t→0 ,F t→1 Predicted optical flow dotted line F t→0 ,F t→1 , F t→0 ,F t→1 Use formula (2) to estimate:

[0019]

[0020] Among them, F 0→1Indicates the optical flow of the pixel from T = 0 to T = 1, F 1→0 Represents the optical flow of pixels transforming from T=1 to T=0, represents the optical flow of pixels from T = t to T = 0, Represents the optical flow of pixels transformed from T = t to T = 1;

[0021] Only one view map is generated through the network, and the other is directly generated by formula (3). The result is input into the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is realized as shown in formula (4);

[0022] V t←0 =1-V t←1 (3)

[0023]

[0024] Among them, V t←0 ,V t←1 Represents the view map ∈ [0,1], z = (1-t) V t←0 +tV t←1 represents the normalization factor;

[0025] Step 1.2: Use the depth information to correct the view map and synthesize the prediction frame;

[0026] The input two frames are I0 and I1, and the generated prediction frame is I t , put the two frames of data, i.e., I0 and I1, into the pre-trained network to obtain their respective depths pred_depth∈(0,+∞), and normalize the result using formula (5) to obtain the normalized depth depth. Subtract the depth of the two frames of data using formula (6) to obtain the depth difference dif between the two frames. When the depths of the two frames change, it means that occlusion has occurred in the area. This value is used to modify the view map.

[0027]

[0028] dif = depth0 - depth1 (6)

[0029] Among them, depth0 represents the depth of I0 after normalization, and depth1 represents the depth of I1 after normalization;

[0030] By using formula (7), when dif is small, the value is directly set to zero, and by using formula (8), dif is further processed.

[0031]

[0032] dif = sigmoid(dif) (8)

[0033] Among them, sigmoid is an activation function, and its output ∈(0,1);

[0034] The view map is used to correct equation (4), and the correction coefficient is introduced into the synthetic prediction frame module, that is, equation (4) is modified into equation (9):

[0035]

[0036] The process of the second step is as follows:

[0037] Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames. Sample TRAIN_DATA according to the set sampling value as shown in formula (10). Put the sampled video frames into TRAIN_LOADER.

[0038] TRAIN_LOADER={TRAIN_DATA(i)|i=n*num_interp,n∈N} (10)

[0039] Step 2.2: Group the training data TRAIN_LOADER. Group the TRAIN_LOADER into groups, group three adjacent video frames into one group, and discard the redundant video frames. Then, shuffle the order of each group and input them for training.

[0040] Step 2.3: Randomly intercept TRAIN_LOADER. To reduce training time, the video frame is intercepted with a fixed size (384, 384) and saved as a tensor as the final video frame for training. The interception position is randomly selected.

[0041] The process of step three is as follows:

[0042] Step 3.1: Preprocess TRAIN_LOADER: Subtract a fixed mean from each set of input video frames as shown in formula (11). After subtracting the mean, in order to enable the model to obtain a larger perception field of view, the input video frames are downsampled by 2 times.

[0043] I0,I1,I2=I0-MEAN_PIX,I1-MEAN_PIX,I2-MEAN_PIX (11)

[0044] Wherein, I0, I1, I2 represent the corresponding input video frames, MEAN_PIX represents a fixed value, MEAN_PIX = [109.93, 109.167, 101.455];

[0045] Step 3.2: Put the processed three frames of data into the model training: randomly input the input video frames in the order of T=0, T=1, T=2 or in reverse order, predict the predicted frames at T=t and T=t+1, and then put these two predicted frames into the model to predict the value of T=1 It is used as the output to compare with I2 to calculate the loss value. The value of t is randomly selected from (0,1), and its selection range is as shown in formula (12)

[0046] t∈(1,2,…,num_interp-1)(12).

[0047] The process of step 4 is as follows: Calculate the loss value using the generated prediction frame. The calculation of the loss value consists of five parts, which are defined as follows:

[0048]

[0049] represents the difference between the cyclically reconstructed frame and the true value, which is defined as in equation (14), It indicates the closeness between the hidden intermediate frame and the pseudo intermediate frame, and its definition is as shown in formula (15). represents the size of the perceptual loss, which is represented by the conv4-3 feature Ψ of the VGG-16 model, and its definition is shown in formula (16), It represents the loss caused by the change of the frame according to the predicted optical flow, which is defined as formula (17). Indicates the speed of optical flow change, which is used to enforce smoothness constraints. Its definition is as shown in formula (18):

[0050]

[0051] in, It represents the expert model, which is trained by the Super SloMo model through a large number of high frame rate videos;

[0052]

[0053]

[0054] The size of the loss function is given by and Decide, The loss value is calculated by the cycle consistency method, introducing to provide guidance, and introduced Dropout to enhance the generalization ability of model training.

[0055] The process of step six is ​​as follows:

[0056] Step 6.1: Extract a set of video frames from the test data TEST_DATA, select the first and last video frames of the dataset from TEST_DATA as the input of the model, and the interval of the video frames is test_num_interp.

[0057] Step 6.2: Pad the input test data with zeros to determine whether the input video frame width and height are multiples of 64. If not, pad the edges with 0 and add them at the end.

[0058] Step 6.3: Generate prediction frames through the trained model, input two temporally adjacent video frames each time, and output test_num_interp prediction frames.

[0059] The beneficial effects of the present invention are mainly manifested in: improving the prediction effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a diagram of the classic video frame insertion training structure.

[0061] Figure 2 This is a system structure diagram of the unsupervised video interpolation method based on depth information and random Dropout.

[0062] Figure 3 It is a diagram of the prediction frame generation process (model).

[0063] Figure 4 It is the optical flow approximation graph.

[0064] Figure 5 It is a schematic diagram of the occlusion situation of the prediction frame.

[0065] Figure 6 It is a model training flow chart.

[0066] Figure 7 It is a flow chart of prediction frame generation.

[0067] Figure 8 It is the flowchart of expert model prediction frame generation. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings.

[0069] Reference Figure 2 to Figure 8 , an unsupervised video interpolation method based on depth information and random Dropout, comprising the following steps:

[0070] Step 1: Generate prediction frame through model (I t): Extract two consecutive video frames (I0, I1) from the low frame rate video to be inserted and generate a prediction frame. The specific process of generating the prediction frame is as follows: Figure 3 The function of the model is: input two frames of images I0, I1, and output the interpolated frame I at any time t between the two frames t , expressed as:

[0071]

[0072] in, Indicates the model mode of training;

[0073] The process of step one is as follows:

[0074] Step 1.1: Obtain the optical flow and field of view map of the input video frame through the neural network model, such as Figure 3 As shown in the figure, the training model uses the Super SloMo model. The model is divided into two networks (U-net1 and U-net2) and some data processing (optical flow mapping and synthetic prediction frames). In the optical flow mapping module, g(*) indicates the use of optical flow F t→0 and F t→1 Map the input frame, optical flow (F t→0 ,F t→1 ) The optical flow (F) between two frames generated by U-net1 0→1 ,F 1→0 ) to make an approximate estimate;

[0075] exist Figure 4 In the example, each point represents a pixel. Assuming that the change of optical flow is smooth, the optical flow F between video frames is t→0 ,F t→1 (solid line) predicted optical flow F t→0 ,F t→1 (dashed line), F t→0 ,F t→1 Use formula (2) to estimate:

[0076]

[0077] Among them, F 0→1 Indicates the optical flow of the pixel from T = 0 to T = 1, F 1→0 Represents the optical flow of pixels transforming from T=1 to T=0, represents the optical flow of pixels from T = t to T = 0, Represents the optical flow of pixels transformed from T = t to T = 1;

[0078] Figure 3 U-net 2 in generates optical flow (F t→0 ,F t→1) and viewshed maps (V t←1 ), for the estimation of the view map, since the same position is rarely blocked in both frames, in order to prevent divergence in network training, only one view map is generated through the network, and the other is directly generated by formula (3). The result is input to the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is implemented as shown in formula (4)

[0079] V t←0 =1-V t←1 (3)

[0080]

[0081] Among them, V t←0 ,V t←1 Represents the view map ∈ [0,1], z = (1-t) V t←0 +tV t←1 represents the normalization factor;

[0082] Step 1.2: Use depth information to correct the view map and synthesize the predicted frame. Since the view map is directly generated by the model and there is no direct supervision item to supervise it, its accuracy is difficult to guarantee. Depth information can help confirm the occlusion of objects and can replace some functions of the view map.

[0083] like Figure 5 The object is moving to block the background. The input two frames are I0 and I1, and the generated prediction frame is I t , the dotted frame in I1 has changed due to the movement of the object, and the depth information has also changed. However, the background here does not appear in I0, so in I t For the prediction in the dotted area, the weight of I1 needs to be larger.

[0084] After putting the two frames of data (I0 and I1) into the pre-trained network (depth information extraction in the figure), we get their respective depths pred_depth∈(0,+∞). We normalize the result using formula (5) to get the normalized depth depth. We subtract the depth of the two frames from each other using formula (6) to get the depth difference dif between the two frames. When the depths of the two frames change, it means that occlusion occurs in the area. This value is used to modify the view map.

[0085]

[0086] dif = depth0 - depth1 (6)

[0087] Among them, depth0 represents the depth of I0 after normalization, and depth1 represents the depth of I1 after normalization;

[0088] Since the change in depth when occlusion occurs is not small under normal circumstances, in order to reduce the impact caused by the error, the formula (7) is used to directly set the value to zero when dif is small. Since the change of dif when it is large should not have a great impact on the result of the weight, and dif should ∈ (0, 1), dif is further processed by formula (8).

[0089]

[0090] dif = sigmoid(dif) (8)

[0091] Among them, sigmoid is an activation function, and its output ∈(0,1);

[0092] In addition to the depth information, the view map determines the weight of the video frames generated by the previous and next frames. For example, if the object moves out of the screen, the view map cannot be directly generated based on the depth information. Therefore, the view map is used to correct equation (4). The correction coefficient is introduced into the synthetic prediction frame module, that is, equation (4) is modified to equation (9):

[0093]

[0094] Step 2. Preprocess the training set for the training model. Input the high frame rate training set TRAIN_DATA, the number of interpolated frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER. The process is as follows:

[0095] Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames: Sample TRAIN_DATA according to the set sampling value as shown in formula (10) and put the sampled video frames into TRAIN_LOADER.

[0096] TRAIN_LOADER={TRAIN_DATA(i)|i=n*num_interp,n∈N} (10)

[0097] Step 2.2: Group training data TRAIN_LOADER: Group TRAIN_LOADER, group three adjacent video frames into one group, discard the redundant video frames directly, and then input the shuffled order of each group for training.

[0098] Step 2.3: Randomly cut the TRAIN_LOADER: To reduce the training time, the video frame is cut into a fixed size (384, 384) and saved as a tensor as the final video frame for training. The cut position is randomly selected.

[0099] Step 3: Generate prediction frames through the model. Input a set of data (I0, I1, I2) in TRAIN_LOADER, the model to be trained, and output the prediction frame The process of training the model is as follows Figure 6 , the process is as follows:

[0100] Step 3.1: Preprocess TRAIN_LOADER: Subtract a fixed mean from each group of input video frames as shown in formula (11). After subtracting the mean, in order to enable the model to obtain a larger perception field of view, the input video frames are downsampled by 2 times.

[0101] I0,I1,I2=I0-MEAN_PIX,I1-MEAN_PIX,I2-MEAN_PIX (11)

[0102] Among them, I0, I1, I2 represent the corresponding input video frames, MEAN_PIX represents a fixed value, MEAN_PIX = [109.93, 109.167, 101.455].

[0103] Step 3.2: Put the processed three frames of data into the model training: Input the video frames as follows Figure 7 Input T=0, T=1, T=2 in order or in reverse order (randomly), and predict the predicted frames at T=t and T=t+1. Then put these two predicted frames into the model to predict the value of T=1 It is used as output and compared with I2 to calculate the loss value. Figure 7 In that is Figure 3 The value of t is randomly selected from (0,1), and its selection range is as shown in formula (12).

[0104] t∈(1,2,…,num_interp-1)(12)

[0105] Step 4: Update the loss value calculation of the model. The loss value calculation is performed using the generated prediction frame. The loss value calculation consists of five parts, which are defined as follows:

[0106]

[0107] represents the difference between the cyclically reconstructed frame and the true value, and is defined as in Equation (14). It represents the closeness between the hidden intermediate frame and our pseudo intermediate frame, and its definition is as shown in formula (15). It represents the size of the perceptual loss, which is represented by the conv4-3 feature (Ψ) of the VGG-16 model and is defined as shown in formula (16). It represents the loss after the frame is changed according to the predicted optical flow, and its definition is as shown in formula (17). It indicates the speed of optical flow change and is used to enforce smoothness constraints. Its definition is as shown in formula (18).

[0108]

[0109] in,

[0110] It represents the expert model, which is trained by the Super SloMo model through a large number of high frame rate videos.

[0111]

[0112] The size of the loss function is mainly determined by and Decide. The loss value is calculated by the cycle consistency method. This method can generate a loss function even when there is only a low frame rate video, but it also loses some basic facts. To provide guidance. While guiding the model's basic facts, it will also introduce some features that are not needed by the current training video, and even lead to poor training results. Because basic facts are ubiquitous, and some features only exist in certain nodes, Dropout is introduced to enhance the generalization ability of model training, so as to remove Some undesirable features.

[0113] Dropout technology is a regularization method. Its main principle is to randomly ignore the output of some neurons with a certain probability during neural network training, that is, to set the output to 0. This method can effectively reduce the overfitting phenomenon of the model and improve the generalization ability of the model, that is, it can reduce However, the traditional method of adding the Dropout layer to the training model is to remove unnecessary features. The above characteristics will also weaken The features learned in The features in are what need to be learned, so the Dropout layer is added to the expert model. The process of generating prediction frames is as follows Figure 8The expert model uses a two-layer network design. The second layer U-Net2 generates residuals, so it is more stable and less prone to gradient explosion. Therefore, the Dropout layer is placed after U-Net2.

[0114] Step 5: Update the model using the loss value. The model is updated using the gradient update method, and the specific optimizer is Adam. Call optimizer.step() to update the parameters for each scene of training data. Training all training video frames once is counted as one cycle. Calculate the sum of the training loss values ​​of each scene in a cycle, epoch_loss, and use this value as the basis for updating the learning rate. The update method uses ReduceLROnPlateau. When the learning rate is less than 1e-7, the training ends and the model is saved.

[0115] Step 6: Use the test data to test the trained model. The specific implementation method is as follows:

[0116] Step 6.1: Extract a set of video frames from the test data TEST_DATA. Select the first and last frames (interval test_num_interp) of the dataset from TEST_DATA as the input of the model.

[0117] Step 6.2: Pad the input test data with zeros. Determine whether the width and height of the input video frame are multiples of 64. If not, pad the edges with 0 (paste at the back). For example, a video frame with a resolution of 1080*720 is padded to 1080*768.

[0118] Step 6.3: Generate a prediction frame using the trained model. The process of generating a prediction frame is as follows: Figure 2 As shown in the figure, each time two temporally adjacent video frames are input, test_num_interp predicted frames are output.

[0119] The unsupervised video interpolation model training method of this embodiment introduces an expert model, which causes the trained model to overfit. The method of adding a Dropout layer to the expert model of the present invention eliminates the overfitting phenomenon of the unique features in the expert model. At the same time, the present invention introduces a depth information method to predict the occlusion situation through depth information, further improving the accuracy of the predicted frame. The advantage of using the method of the present invention is that a video interpolation model can still be trained even when it is completely impossible to obtain relevant high frame rate videos, and it can have higher accuracy.

[0120] In order to verify the performance of the method of the present invention, a comparative experiment was carried out. The corresponding parameter of the loss value is set to λ rc =0.8,λ rp=0.8,λ p =0.05,λ w =0.4,λ s =1. The model parameter optimization algorithm uses Adam. The learning rate is updated using ReduceLROnPlateau, where mode = 'min', patience = 3, facter = 0.1. The rest of the parameters are default.

[0121] The dataset uses part0 of REDS (120fps) and the mobile phone shooting part of MSU (60fps), named MSU_4 in this paper. The first num_interp+1 frames are taken as the test set TEST_DATA, and the rest of the data are sampled at 30fps as the training dataset TRAIN_DATA for training. The num_interp values ​​of REDS and MSU_4 are 3 and 1 respectively.

[0122] In experiment 1, a Dropout layer is added to the expert model when training the model. The training results of the dataset with the Dropout layer added are shown in Table 1. The PSNR value is the average of the results of multiple runs. From the Train method, it can be seen that the traditional Dropout method, that is, the method of placing the Dropout layer in the training model, has a reduced effect because it also removes the features brought by the relevant video loss value. The results of the two datasets have demonstrated the role of adding a Dropout layer to the expert model. Among them, the results of the REDS dataset are relatively stable. The results of the MSU_4 dataset with the Dropout layer added to the expert model have been greatly improved but are relatively unstable. This is because the features of the dataset are quite different from those of the expert model, and the Dropout layer cannot stably remove the features on the expert model. By comparing the results of two different datasets, it can be found that the optimal Dropout layer parameters of the two datasets are different. This is because the two datasets contain different features, and the amount of features that need to be removed on the expert model is also different.

[0123]

[0124] Table 1

[0125] Baseline means that the expert model is directly evaluated. org means the solution proposed by Reda et al. Train:Dropout=0.2 means that the Dropout layer is placed in the training model with the parameter set to 0.2. Tea:Dropout=0.2 means that the Dropout layer is placed in the expert model with the parameter set to 0.2. ΔPSNRmax means the maximum difference in PSNR in each training result.

[0126] In Experiment 2, the model trained in Experiment 1 was used as the expert model, and depth information was introduced to further train the model. The results are shown in Table 2, which shows the training results of the view map correction of the dataset. Since the variance of the training results directly starting from 1e-4 is large, and the method of introducing depth information has a small improvement, in order to prove the role of depth information, in this experiment, the model with good training results in different ways was directly used for initialization, so the initialization of the learning rate was set to a small value. From the results in the table, it can be seen that the method of correcting the view map through depth information can stably improve the training effect.

[0127]

[0128] Table 2

[0129] Where depth means adding depth information to train the model

[0130] Experiment 3: Video interpolation result test. The trained model was tested. The results show that the predicted frames generated by the method proposed in the present invention are significantly improved.

[0131] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. An unsupervised video interpolation method based on depth information and random Dropout, characterized in that: The method comprises the following steps: Step 1: Generate prediction frame I through model t : Extract two consecutive video frames (I0, I1) from the low frame rate video to be inserted and generate a predicted frame. The function of the model is to input two frames of images I0, I1 and output the inserted frame I0 at any time t between the two frames. t , expressed as: in, Indicates the model mode of training; Step 2: Preprocess the training set of the training model, input the high frame rate training set TRAIN_DATA, the number of interpolation frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER; Step 3: Generate a prediction frame through the model, input a set of data (I0, I1, I2) in TRAIN_LOADER, the model to be trained, and output the prediction frame Step 4: Update the loss value calculation of the model; Step 5. Update the model with the obtained loss value: The model is updated by gradient update, and the optimizer is Adam. After each training scene, optimizer.step() is called to update the parameters. Training all the video frames is counted as one cycle, and the sum of the training loss values ​​of each scene in one cycle is calculated. This value is used as the basis for updating the learning rate. The updating method is ReduceLROnPlateau. When the learning rate is less than 1e-7, the training ends and the model is saved. Step 6: Use the test data to test the trained model.

2. The unsupervised video frame insertion method based on depth information and random Dropout as claimed in claim 1, characterized in that: The process of step one is as follows: Step 1.1: Obtain the optical flow and field of view map of the input video frame through the neural network model. In the optical flow mapping module, g(*) represents the use of optical flow F t→0 and F t→1 Map the input frame, optical flow (F t→0 ,F t→1 ) The optical flow (F) between two frames generated by U-net1 0→1 ,F 1→0 ) to make an approximate estimate; Assuming that the change of optical flow is smooth, the optical flow solid line F between video frames t→0 ,F t→1 Predicted optical flow dotted line F t→0 ,F t→1 , F t→0 ,F t→1 Use formula (2) to estimate: Among them, F 0→1 Indicates the optical flow of the pixel from T = 0 to T = 1, F 1→0 Represents the optical flow of pixels transforming from T=1 to T=0, represents the optical flow of pixels from T = t to T = 0, Represents the optical flow of pixels transformed from T = t to T = 1; Only one view map is generated through the network, and the other is directly generated by formula (3). The result is input into the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is realized as shown in formula (4); V t←0 =1-V t←1 (3) Among them, V t←0 ,V t←1 Represents the view map ∈ [0,1], z = (1-t) V t←0 +tV t←1 represents the normalization factor; Step 1.2: Use the depth information to correct the view map and synthesize the prediction frame; The input two frames are I0 and I1, and the generated prediction frame is I t , put the two frames of data, i.e., I0 and I1, into the pre-trained network to obtain their respective depths pred_depth∈(0,+∞), and normalize the result using formula (5) to obtain the normalized depth depth. Subtract the depth of the two frames of data using formula (6) to obtain the depth difference dif between the two frames. When the depths of the two frames change, it means that occlusion has occurred in the area. This value is used to modify the view map. dif = depth0 - depth1 (6) Among them, depth0 represents the depth of I0 after normalization, and depth1 represents the depth of I1 after normalization; By using formula (7), when dif is small, the value is directly set to zero, and by using formula (8), dif is further processed. dif = sigmoid(dif) (8) Among them, sigmoid is an activation function, and its output ∈(0,1); The view map is used to correct equation (4), and the correction coefficient is introduced into the synthetic prediction frame module, that is, equation (4) is modified into equation (9):

3. The unsupervised video interpolation method based on depth information and random Dropout according to claim 1 or 2, characterized in that: The process of the second step is as follows: Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames. Sample TRAIN_DATA according to the set sampling value as shown in formula (10). Put the sampled video frames into TRAIN_LOADER. TRAIN_LOADER={TRAIN_DATA(i)|i=n*num_interp,n∈N}(10) Step 2.2: Group the training data TRAIN_LOADER. Group the TRAIN_LOADER into groups, group three adjacent video frames into one group, and discard the redundant video frames. Then, shuffle the order of each group and input them for training. Step 2.3: Randomly intercept TRAIN_LOADER. To reduce training time, the video frame is intercepted with a fixed size (384, 384) and saved as a tensor as the final video frame for training. The interception position is randomly selected.

4. The unsupervised video frame insertion method based on depth information and random Dropout as claimed in claim 3, characterized in that: The process of step three is as follows: Step 3.1: Preprocess TRAIN_LOADER: Subtract a fixed mean from each set of input video frames as shown in formula (11). After subtracting the mean, in order to enable the model to obtain a larger perception field of view, the input video frames are downsampled by 2 times. I0,I1,I2=I0-MEAN_PIX,I1-MEAN_PIX,I2-MEAN_PIX(11) Wherein, I0, I1, I2 represent the corresponding input video frames, MEAN_PIX represents a fixed value, MEAN_PIX = [109.93, 109.167, 101.455]; Step 3.2: Put the processed three frames of data into the model training: randomly input the input video frames in the order of T=0, T=1, T=2 or in reverse order, predict the predicted frames at T=t and T=t+1, and then put these two predicted frames into the model to predict the value of T=1 It is used as the output to compare with I2 to calculate the loss value. The value of t is randomly selected from (0,1), and its selection range is as shown in formula (12) t∈(1,2,…,num_interp-1)(12).

5. The unsupervised video frame insertion method based on depth information and random Dropout as claimed in claim 4, characterized in that: The process of step 4 is as follows: Calculate the loss value using the generated prediction frame. The calculation of the loss value consists of five parts, which are defined as follows: represents the difference between the cyclically reconstructed frame and the true value, which is defined as in equation (14), It indicates the closeness between the hidden intermediate frame and the pseudo intermediate frame, and its definition is as shown in formula (15). represents the size of the perceptual loss, which is represented by the conv4-3 feature Ψ of the VGG-16 model, and its definition is shown in formula (16), It represents the loss caused by the change of the frame according to the predicted optical flow, which is defined as formula (17). Indicates the speed of optical flow change, which is used to enforce smoothness constraints. Its definition is as shown in formula (18): in, It represents the expert model, which is trained by the Super SloMo model through a large number of high frame rate videos; The size of the loss function is given by and Decide, The loss value is calculated by the cycle consistency method, introducing to provide guidance, and introduced Dropout to enhance the generalization ability of model training.

6. The unsupervised video frame insertion method based on depth information and random Dropout according to claim 1 or 2, characterized in that: The process of step six is ​​as follows: Step 6.1: Extract a set of video frames from the test data TEST_DATA, select the first and last video frames of the dataset from TEST_DATA as the input of the model, and the interval of the video frames is test_num_interp; Step 6.2: Pad the input test data with zeros to determine whether the input video frame width and height are multiples of 64. If not, pad the edges with 0 and add them at the end. Step 6.3: Generate prediction frames through the trained model, input two temporally adjacent video frames each time, and output test_num_interp prediction frames.

Images