Unsupervised video frame interpolation method based on depth information and random dropout

Abstract

The application discloses an unsupervised video frame interpolation method based on depth information and random Dropout, and comprises the following steps: step one, generating a predicted frame I through a model t ; step two, preprocessing a training set for training the model; step three, generating a predicted frame through the model; step four, updating loss value calculation of the model; step five, updating the model through the obtained loss value; and step six, testing the trained model by using test data. The application adds a Dropout layer to an expert model to reduce the influence caused by overfitting. Moreover, the application introduces depth information to judge the front-back relationship of objects, solves the blur problem caused by object occlusion, and further improves the prediction effect of the model.

Description

Technical Field

[0001] The present invention belongs to the field of video processing, and in particular relates to an unsupervised video frame insertion method based on depth information and random Dropout. Background Art

[0002] In the field of video processing, in order to improve the smoothness of the video and the viewing experience, it is necessary to insert frames into the original video. Today's video interpolation methods are mainly based on the training of neural network models. This requires high-frequency videos for supervised training models. With the development of society, the demand for high-frame rate videos in different scenarios continues to increase. At the same time, obtaining a large number of high-frame rate videos requires high costs (storage space, power consumption, etc.) and equipment that can obtain high-frame rate videos. How to train a video interpolation model without relevant high-frame rate video acquisition methods has become a hot issue. Currently, in order to address the problem of high model training costs, the method of using a large amount of data to train a model is generally used to improve its generalization ability. However, this method also causes the model to not reflect the characteristics of the current video that needs to be inserted well. For this reason, a method of using unsupervised technology to train the model has emerged.

[0003] Video interpolation (VI) is a method used in the field of computer science to increase the frame rate of video. This method is mainly based on convolutional neural networks. It uses algorithms to automatically learn the movement and change patterns between video frames, thereby generating high-quality intermediate frames. When high-frame-rate videos cannot be obtained, unsupervised video interpolation technology is required. Unsupervised video interpolation technology does not require a large number of labeled data sets for training. It only generates intermediate frames by automatically learning the inherent patterns between video frames. The current video interpolation technology is mainly trained through high-frame-rate video supervision, and its methods are as follows. Figure 1 As shown in the figure, first, a low frame rate (Low FPS) video is input; then, the bidirectional optical flow network is used to generate the optical flow information of the two frames and calculate the intermediate optical flow information; then, the optical flow information is used to calculate the intermediate interpolated frame image to obtain a high frame rate (High FPS) video; finally, the high frame rate video is used for supervision to calculate the loss value to update the model parameters, and finally a model that can perform video interpolation is obtained.

[0004] Currently, many experts and scholars are researching video interpolation methods. In 2018, Jiang et al. achieved high-quality intermediate frame estimation by training deep neural networks. In 2024, researchers from Shanghai Jiao Tong University and other institutions proposed a new perception-oriented video frame interpolation paradigm that effectively addresses the inaccurate motion estimation and supervision signal alignment issues. However, when relevant high-frame-rate video for supervision is unavailable, unsupervised training methods are required. However, these methods lack some basic fundamentals. To this end, Reda et al. proposed using an expert model for pseudo-supervision (PS). However, the expert model used in this method is not trained on the same type of video, and the model may contain some unnecessary features. Therefore, models trained using pseudo-supervision methods are subject to some overfitting. Furthermore, since the final synthesis requires a view map to determine whether occlusion occurs, the method proposed by Reda et al. does not directly monitor the view map for loss, which can exacerbate background blur at object edges. In summary, how to reduce the influence of the expert model on the training results during unsupervised training of the interpolation model and strengthen the judgment of image occlusion problems is an important issue affecting the effectiveness of the video interpolation model. Summary of the Invention

[0005] In order to overcome the shortcomings of existing technologies and address the problem of video interpolation in an unsupervised environment, the present invention provides an unsupervised video interpolation method based on depth information and random Dropout. A Dropout layer is added to the expert model to reduce the impact of overfitting. The present invention also proposes introducing depth information to judge the fore-and-aft relationship of objects, solving the blur problem caused by object occlusion, and further improving the prediction effect of the model.

[0006] The technical solution adopted by the present invention to solve its technical problem is:

[0007] An unsupervised video interpolation method based on depth information and random dropout includes the following steps:

[0008] Step 1: Generate prediction frame I through model t : Extract two consecutive video frames (I0, I1) from the low frame rate video to be inserted and generate a predicted frame. The function of the model is to input two frames of image I0, I1 and output the inserted frame I at any time t between the two frames. t , expressed as:

[0009]

[0010] in, Indicates the training model mode;

[0011] Step 2: Preprocess the training set of the training model, input the high frame rate training set TRAIN_DATA, the number of interpolated frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER;

[0012] Step 3: Generate a prediction frame through the model, input a set of data (I0, I1, I2) in TRAIN_LOADER, the model to be trained, and output the prediction frame

[0013] Step 4: Update the loss value calculation of the model;

[0014] Step 5. Update the model using the obtained loss value: The model is updated using the gradient update method, the optimizer is Adam, and optimizer.step() is called for each training scene to update the parameters; training all video frames once is counted as one cycle, and the sum of the training loss values ​​of each scene in a cycle is calculated as epoch_loss. This value is used as the basis for updating the learning rate. The update method is ReduceLROnPlateau. When the learning rate is less than 1e-7, the training ends and the model at this time is saved;

[0015] Step 6: Use the test data to test the trained model.

[0016] Furthermore, the process of step one is as follows:

[0017] Step 1.1: Obtain the optical flow and view map of the input video frame through the neural network model. In the optical flow mapping module, g(*) represents the use of optical flow F t→0 and F t→1 Map the input frame, optical flow (F t→0 ,F t→1 ) The optical flow (F) between the two frames generated by U-net1 0→1 ,F 1→0 ) to make an approximate estimate;

[0018] Assuming that the change of optical flow is smooth, the optical flow solid line F between video frames t→0 ,F t→1 Predicted optical flow dotted line F t→0 ,F t→1 , F t→0 ,F t→1 Use formula (2) to estimate:

[0019]

[0020] Among them, F 0→1Indicates the optical flow of pixels transformed from T = 0 to T = 1, F 1→0 Represents the optical flow of pixels transformed from T=1 to T=0, Represents the optical flow of pixels transformed from T = t to T = 0, Represents the optical flow of pixels transformed from T = t to T = 1;

[0021] Only one view map is generated through the network, and the other is directly generated by formula (3). The result is input into the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is realized as shown in formula (4);

[0022] V t←0 =1-V t←1 (3)

[0023]

[0024] Among them, V t←0 ,V t←1 Represents the view map ∈ [0,1], z = (1-t) V t←0 +tV t←1 represents the normalization factor;

[0025] Step 1.2: Use depth information to correct the view map and synthesize the predicted frame;

[0026] The two frames of input data are I0 and I1, and the generated prediction frame is I t , put the two frames of data, i.e. I0 and I1, into the pre-trained network to obtain their respective depths pred_depth∈(0,+∞), and normalize the result by formula (5) to obtain the normalized depth depth. Subtract the depth of the two frames by formula (6) to obtain the depth difference dif of the two frames. When the depth of the two frames changes, it means that there is occlusion in the area. The view map is modified by this value.

[0027]

[0028] dif = depth0 - depth1 (6)

[0029] Among them, depth0 represents the depth of I0 after normalization, and depth1 represents the depth of I1 after normalization;

[0030] By using formula (7), when dif is small, the value is directly set to zero, and by using formula (8), dif is further processed.

[0031]

[0032] dif = sigmoid(dif) (8)

[0033] Among them, sigmoid is an activation function, and its output ∈(0,1);

[0034] The view map is used to modify Equation (4), and the correction coefficient is introduced into the synthetic prediction frame module, that is, Equation (4) is modified into Equation (9)

[0035]

[0036] The process of the second step is as follows:

[0037] Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames. Sample TRAIN_DATA according to the set sampling value as shown in formula (10). Put the sampled video frames into TRAIN_LOADER.

[0038] TRAIN_LOADER={TRAIN_DATA(i)|i=n*num_interp,n∈N} (10)

[0039] Step 2.2: Group the training data TRAIN_LOADER. Group the TRAIN_LOADER into groups of three adjacent video frames. Discard the redundant video frames. Then, shuffle the order of each group and input them for training.

[0040] Step 2.3: Randomly intercept the TRAIN_LOADER. To reduce training time, the video frame is intercepted with a fixed size (384,384) and saved as a tensor as the final video frame for training. The interception position is randomly selected.

[0041] The process of step three is as follows:

[0042] Step 3.1: Preprocessing TRAIN_LOADER: Subtract a fixed mean from each set of input video frames as shown in formula (11). After subtracting the mean, in order to enable the model to obtain a larger perception field of view, the input video frames are downsampled by 2 times.

[0043] I0,I1,I2=I0-MEAN_PIX,I1-MEAN_PIX,I2-MEAN_PIX (11)

[0044] Where I0, I1, I2 represent the corresponding input video frames, MEAN_PIX represents a fixed value, MEAN_PIX = [109.93, 109.167, 101.455];

[0045] Step 3.2: Put the processed three frames of data into the model training: randomly input the input video frames in the order of T=0, T=1, T=2 or in reverse order, predict the predicted frames at T=t and T=t+1, and then put these two predicted frames into the model to predict the value of T=1 It is used as the output and compared with I2 to calculate the loss value. The value of t is randomly selected from (0,1), and its selection range is as shown in formula (12)

[0046] t∈(1,2,…,num_interp-1)(12).

[0047] The process of step 4 is as follows: Calculate the loss value using the generated prediction frame. The calculation of the loss value consists of five parts, which are defined as follows:

[0048]

[0049] represents the difference between the cyclically reconstructed frame and the true value, which is defined as in Equation (14), Indicates the closeness between the hidden intermediate frame and the pseudo intermediate frame, which is defined as formula (15), represents the size of the perceptual loss, which is represented by the conv4-3 feature Ψ of the VGG-16 model, and is defined as shown in formula (16), It represents the loss caused by the change of the frame according to the predicted optical flow, which is defined as formula (17). Indicates the speed of optical flow change, which is used to enforce smoothness constraints. It is defined as in formula (18).

[0050]

[0051] in, The expert model is trained by the Super SloMo model through a large number of high-frame-rate videos.

[0052]

[0053]

[0054] The magnitude of the loss function is given by and Decide, The loss value is calculated by the cycle consistency method, which introduces to provide guidance, and introduced Dropout to enhance the generalization ability of model training.

[0055] The process of step six is ​​as follows:

[0056] Step 6.1: Extract a set of video frames from the test data TEST_DATA, select the first and last video frames of the dataset from TEST_DATA as the input of the model, and the interval of the video frames is test_num_interp.

[0057] Step 6.2: Pad the input test data with zeros to determine whether the width and height of the input video frame are multiples of 64. If not, pad the edges with 0 and add them at the end.

[0058] Step 6.3: Generate prediction frames through the trained model, input two temporally adjacent video frames each time, and output test_num_interp prediction frames.

[0059] The beneficial effects of the present invention are mainly manifested in: improving the prediction effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a diagram of the classic video frame insertion training structure.

[0061] Figure 2 This is a system structure diagram of the unsupervised video interpolation method based on depth information and random Dropout.

[0062] Figure 3 It is a model diagram of the prediction frame generation process.

[0063] Figure 4 is the optical flow approximation graph.

[0064] Figure 5 It is a schematic diagram of the occlusion situation of the predicted frame.

[0065] Figure 6 It is a model training flow chart.

[0066] Figure 7 This is a flow chart of prediction frame generation.

[0067] Figure 8 It is the flowchart of expert model prediction frame generation. DETAILED DESCRIPTION

[0068] The present invention will be further described below with reference to the accompanying drawings.

[0069] Reference Figures 2 to 8 , an unsupervised video interpolation method based on depth information and random dropout, comprising the following steps:

[0070] Step 1: Generate prediction frame through model (I t): Extract two consecutive video frames (I0, I1) from the low frame rate video to be inserted and generate a prediction frame. The specific process of generating the prediction frame is as follows Figure 3 The function of the model is to input two frames of image I0, I1 and output the interpolated frame I at any time t between the two frames. t , expressed as:

[0071]

[0072] in, Indicates the training model mode;

[0073] The process of step one is as follows:

[0074] Step 1.1: Obtain the optical flow and view map of the input video frame through the neural network model, such as Figure 3 As shown, the training model uses the Super SloMo model. The model is divided into two networks (U-net1 and U-net2) and some data processing (optical flow mapping and synthetic prediction frames). In the optical flow mapping module, g(*) indicates the use of optical flow F t→0 and F t→1 Map the input frame, optical flow (F t→0 ,F t→1 ) The optical flow (F) between the two frames generated by U-net1 0→1 ,F 1→0 ) to make an approximate estimate;

[0075] exist Figure 4 In the example, each point represents a pixel. Assuming that the change of optical flow is smooth, the optical flow F between video frames is used to calculate the optical flow. t→0 ,F t→1 (solid line) predicted optical flow F t→0 ,F t→1 (dashed line), F t→0 ,F t→1 Use formula (2) to estimate:

[0076]

[0077] Among them, F 0→1 Indicates the optical flow of pixels transformed from T = 0 to T = 1, F 1→0 Represents the optical flow of pixels transformed from T=1 to T=0, Represents the optical flow of pixels transformed from T = t to T = 0, Represents the optical flow of pixels transformed from T = t to T = 1;

[0078] Figure 3 U-net 2 in generates optical flow (F t→0 ,F t→1) and viewshed maps (V t←1 ), for the estimation of the view map, since the same position is rarely blocked in both frames, in order to prevent divergence in network training, only one view map is generated through the network, and the other is directly generated by formula (3). The result is input to the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is realized as formula (4)

[0079] V t←0 =1-V t←1 (3)

[0080]

[0081] Among them, V t←0 ,V t←1 Represents the view map ∈ [0,1], z = (1-t) V t←0 +tV t←1 represents the normalization factor;

[0082] Step 1.2: Use depth information to correct the view map and synthesize the predicted frame. Since the view map is directly generated by the model and there is no direct supervision, its accuracy is difficult to guarantee. Depth information can help confirm the occlusion of objects and can replace some of the functions of the view map.

[0083] like Figure 5 The occlusion of the background by the moving object. The two frames of input data are I0 and I1, and the generated prediction frame is I t In I1, the dotted box has changed due to the movement of the object, and the depth information has also changed. However, the background here does not have the corresponding content in I0, so in I t For the prediction in the dotted area, the weight of I1 needs to be larger.

[0084] After putting the two frames of data (I0 and I1) into the pre-trained network (depth information extraction in the figure), we get their respective depths pred_depth∈(0,+∞). We normalize the result by formula (5) to get the normalized depth depth. We subtract the depth of the two frames by formula (6) to get the depth difference dif between the two frames. When the depth of the two frames changes, it means that there is occlusion in the area. We modify the view map by using this value.

[0085]

[0086] dif = depth0 - depth1 (6)

[0087] Among them, depth0 represents the depth of I0 after normalization, and depth1 represents the depth of I1 after normalization;

[0088] Since the change in depth when occlusion occurs is not small under normal circumstances, in order to reduce the impact caused by the error, the formula (7) is used to set the value to zero when dif is small. Since the change in dif when it is large should not have a great impact on the weight result, and dif should be ∈ (0, 1), dif is further processed by formula (8).

[0089]

[0090] dif = sigmoid(dif) (8)

[0091] Among them, sigmoid is an activation function, and its output ∈(0,1);

[0092] In addition to depth information, the view map determines the weight of the video frames generated by the previous and next frames. For example, if an object moves out of the picture, the view map cannot be directly generated based on the depth information. Therefore, the view map is used to correct Equation (4). The correction coefficient is introduced into the synthetic prediction frame module, that is, Equation (4) is modified to Equation (9)

[0093]

[0094] Step 2. Preprocess the training set for the training model. Input the high frame rate training set TRAIN_DATA, the number of interpolated frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER. The process is as follows:

[0095] Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames: Sample TRAIN_DATA according to the set sampling value as shown in formula (10) and put the sampled video frames into TRAIN_LOADER.

[0096] TRAIN_LOADER={TRAIN_DATA(i)|i=n*num_interp,n∈N} (10)

[0097] Step 2.2: Group training data TRAIN_LOADER: Group TRAIN_LOADER, group three adjacent video frames into one group, discard the extra video frames directly, and then input the shuffled order of each group for training.

[0098] Step 2.3: Randomly cut the train_loader: To reduce training time, cut the video frame into a fixed size (384,384) and save it as a tensor as the final video frame for training. The cut position is randomly selected.

[0099] Step 3: Generate prediction frames through the model. Input a set of data (I0, I1, I2) in TRAIN_LOADER, the model to be trained, and output the prediction frame. The process of training the model is as follows Figure 6 , the process is as follows:

[0100] Step 3.1: Preprocess the train_loader: Subtract a fixed mean from each set of input video frames as shown in Equation (11). After subtracting the mean, the input video frames are downsampled by a factor of 2 to allow the model to obtain a larger receptive field of view.

[0101] I0,I1,I2=I0-MEAN_PIX,I1-MEAN_PIX,I2-MEAN_PIX (11)

[0102] Among them, I0, I1, I2 represent the corresponding input video frames, MEAN_PIX represents a fixed value, MEAN_PIX = [109.93, 109.167, 101.455].

[0103] Step 3.2: Put the processed three frames of data into model training: Input video frames as follows Figure 7 Input T=0, T=1, T=2 in order or in reverse order (randomly), and predict the predicted frames at T=t and T=t+1. Then put these two predicted frames into the model to predict the value of T=1 Compare it with I2 as the output to calculate the loss value. Figure 7 in that is Figure 3 The value of t is randomly selected from (0,1), and its selection range is as shown in formula (12).

[0104] t∈(1,2,…,num_interp-1)(12)

[0105] Step 4: Update the loss value calculation of the model. The loss value calculation is performed using the generated prediction frame. The loss value calculation consists of five parts, which are defined as follows:

[0106]

[0107] It represents the difference between the cyclically reconstructed frame and the true value, and is defined as in Equation (14). It represents the closeness between the hidden intermediate frame and our pseudo intermediate frame, and is defined as in Equation (15). represents the size of the perceptual loss, which is represented by the conv4-3 feature (Ψ) of the VGG-16 model and is defined as shown in Equation (16). It represents the loss caused by the change of the frame according to the predicted optical flow, and its definition is as shown in formula (17). Indicates the speed of optical flow change, which is used to enforce smoothness constraints. It is defined as in Equation (18).

[0108]

[0109] in,

[0110] It represents the expert model, which is trained by the Super SloMo model through a large number of high frame rate videos.

[0111]

[0112] The size of the loss function is mainly determined by and Decide. The loss value is calculated by the cycle consistency method. This method can generate the loss function even when there is only a low frame rate video, but it also loses some basic facts. For this purpose, we introduce To provide guidance. While guiding the model's basic facts, it also introduces some features that are not needed by the current training video, and may even lead to poor training results. Because basic facts are ubiquitous, and some features only exist in certain nodes, Dropout is introduced to enhance the generalization ability of model training, so as to remove Some unnecessary features.

[0113] Dropout technology is a regularization method. Its main principle is to randomly ignore the output of some neurons with a certain probability during neural network training, that is, to set the output to 0. This method can effectively reduce the overfitting phenomenon of the model and improve the generalization ability of the model, that is, it can reduce However, the traditional method of adding the Dropout layer to the training model is not very effective in removing The above characteristics will also weaken The features learned in . The features in are what need to be learned, so the Dropout layer is added to the expert model. The process of generating prediction frames is as follows Figure 8The expert model uses a two-layer network design. The second layer, U-Net2, generates residuals, making it more stable and less prone to gradient explosion. Therefore, the Dropout layer is placed after U-Net2.

[0114] Step 5: Update the model using the resulting loss. This model update uses a gradient update method, using the Adam optimizer. After each training episode, call optimizer.step() to update the parameters. Training all video frames once is counted as one epoch. The sum of the training loss values ​​for each episode in an epoch is calculated, and this value is used as the basis for updating the learning rate. The update method uses ReduceLROnPlateau. Training ends when the learning rate is less than 1e-7, and the current model is saved.

[0115] Step 6: Use the test data to test the trained model. The specific implementation is as follows:

[0116] Step 6.1: Extract a set of video frames from the test data TEST_DATA. Select the first and last frames (interval test_num_interp) of the dataset from TEST_DATA as the input of the model.

[0117] Step 6.2: Pad the input test data with zeros. Check if the width and height of the input video frame are multiples of 64. If not, pad the edges with zeros (add them to the back). For example, a video frame with a resolution of 1080*720 is padded to 1080*768.

[0118] Step 6.3: Generate prediction frames through the trained model. The process of generating prediction frames is as follows Figure 2 As shown in Figure 2. Each time two temporally adjacent video frames are input, test_num_interp predicted frames are output.

[0119] The unsupervised video interpolation model training method of this embodiment introduces an expert model, which leads to overfitting of the trained model. The method of adding a Dropout layer to the expert model of the present invention eliminates the overfitting phenomenon of the unique features in the expert model. At the same time, the present invention introduces a depth information method to predict occlusion conditions through depth information, further improving the accuracy of the predicted frames. The advantage of using the method of the present invention is that a video interpolation model can still be trained with higher accuracy even when it is completely impossible to obtain relevant high frame rate videos.

[0120] In order to verify the performance of the method of the present invention, a comparative experiment was conducted. The corresponding parameter of the loss value is set to λ rc =0.8,λ rp=0.8,λ p =0.05,λ w =0.4,λ s = 1. Adam is used as the model parameter optimization algorithm. ReduceLROnPlateau is used to update the learning rate, with mode = 'min', patience = 3, and facter = 0.1. All other parameters are left as default.

[0121] The dataset uses part 0 of REDS (120 fps) and the mobile phone capture portion of MSU (60 fps), named MSU_4 in this article. The first num_interp+1 frames are used as the test set TEST_DATA, and the remaining data is sampled at 30 fps as the training dataset TRAIN_DATA for training. The num_interp values ​​for REDS and MSU_4 are 3 and 1, respectively.

[0122] In Experiment 1, a Dropout layer was added to the expert model during model training. The training results of the dataset with the Dropout layer added are shown in Table 1. The PSNR value is the average of the results from multiple runs. The training method shows that the traditional Dropout method, that is, placing the Dropout layer in the training model, reduces the effect because it also removes features brought by the relevant video loss value. The results of both datasets demonstrate the effectiveness of adding a Dropout layer to the expert model. The results of the REDS dataset are relatively stable. The results of adding a Dropout layer to the expert model on the MSU_4 dataset show significant improvement but are less stable. This is because the features of this dataset differ significantly from those of the expert model, and the Dropout layer cannot stably remove the features on the expert model. By comparing the results of the two different datasets, it can be found that the optimal Dropout layer parameters for the two datasets are different. This is because the two datasets contain different features, and the amount of features that need to be removed on the expert model is also different.

[0123]

[0124] Table 1

[0125] Baseline indicates direct evaluation of the expert model. org denotes the approach proposed by Reda et al. Train:Dropout = 0.2 indicates adding a dropout layer to the training model with a parameter of 0.2. Tea:Dropout = 0.2 indicates adding a dropout layer to the expert model with a parameter of 0.2. ΔPSNRmax denotes the maximum PSNR difference between each training result.

[0126] In Experiment 2, the model trained in Experiment 1 was used as the expert model, and depth information was introduced for further training. The results are shown in Table 2, which shows the training results for the corrected view map of the dataset. Because the variance of the results obtained by directly starting training from 1e-4 is large, while the method of introducing depth information has little improvement, to demonstrate the effectiveness of depth information, in this experiment, the model with the best training results using different methods was directly used for initialization. Therefore, the initial learning rate was set to a low level. The results in the table show that correcting the view map using depth information can consistently improve training results.

[0127]

[0128] Table 2

[0129] Where depth means adding depth information to train the model

[0130] Experiment 3: Video Frame Insertion Results Test. The trained model was tested. The results showed that the predicted frames produced by the proposed method were significantly improved.

[0131] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. An unsupervised video interpolation method based on depth information and random dropout, characterized in that: The method comprises the following steps: Step 1: Generate prediction frame through model : Extract two consecutive video frames from the low frame rate video to be inserted ( , ) After input, the prediction frame is generated. The function of the model is to input two frames of image , , output any time between two frames Insertion , expressed as: (1) in, Represents the trained model model; Step 2: Preprocess the training set of the training model, input the high frame rate training set TRAIN_DATA, the number of interpolated frames of the training set num_interp, and output the low frame rate training set TRAIN_LOADER; Step 3: Generate prediction frames through the model and put a set of data in TRAIN_LOADER ( ) input to the model to be trained and output the predicted frame ; Step 4: Update the loss value calculation of the model; Step 5. Update the model using the obtained loss value: The model is updated using the gradient update method, the optimizer is Adam, and optimizer.step() is called for each training scene to update the parameters; training all video frames once is counted as one cycle, and the sum of the training loss values ​​of each scene in a cycle is calculated as epoch_loss. This value is used as the basis for updating the learning rate. The update method is ReduceLROnPlateau. When the learning rate is less than 1e-7, the training ends and the model at this time is saved; Step 6: Use test data to test the trained model; The process of step one is as follows: Step 1.1: Obtain the optical flow and view map of the input video frame through the neural network model. In the optical flow mapping module Indicates the use of optical flow and Mapping the input frame, optical flow ( ) The optical flow between two frames generated by U-net1 ( ) to make an approximate estimate; Assuming that the change of optical flow is smooth, the optical flow solid line between video frames Predicting optical flow dashed lines , Use formula (2) to estimate: (2) in, Indicates the optical flow of pixels transformed from T=0 to T=1, Indicates the optical flow of pixels transformed from T=1 to T=0, Represents the optical flow of pixels transformed from T=t to T=0, Indicates the optical flow of pixels transformed from T=t to T=1; Only one view map is generated through the network, and the other is directly generated by formula (3). The result is input into the synthetic prediction frame module to generate a prediction frame. The generation of the prediction frame is realized as shown in formula (4); (3) (4) in, Representing a viewshed map , represents the normalization factor; Step 1.2: Use depth information to correct the view map and synthesize the predicted frame; The two frames of input data are and , the resulting predicted frame is , the two frames of data before and after, namely and After putting the pre-trained network, get their respective depths , the result is normalized by formula (5) to obtain the normalized depth depth, and the depth difference dif of the two frames is obtained by subtracting the depth of the two frames through formula (6). When the depth of the two frames changes, it means that there is occlusion in the area. The view map is modified by the depth difference. (5) (6) in, After normalization The depth, After normalization Depth; By using formula (7), when dif is small, the value is directly set to zero, and by using formula (8), dif is further processed. (7) (8) Among them, sigmoid is an activation function, whose output ∈(0,1); The view map is used to modify Equation (4), and the correction coefficient is introduced into the synthetic prediction frame module, that is, Equation (4) is modified into Equation (9) (9)； The process of step 4 is as follows: Calculate the loss value using the generated prediction frame. The calculation of the loss value consists of five parts, which are defined as follows: (13) 、 、 、 、 Represent the relevant parameters of the loss value, represents the difference between the cyclically reconstructed frame and the true value, which is defined as in Equation (14), Indicates the closeness between the hidden intermediate frame and the pseudo intermediate frame, which is defined as formula (15), Indicates the size of the perceptual loss, which uses the conv4-3 feature of the VGG-16 model It is defined as shown in formula (16). It represents the loss caused by the change of the frame according to the predicted optical flow, which is defined as formula (17). Indicates the speed of optical flow change, which is used to enforce smoothness constraints. It is defined as in formula (18). (14) (15) in, The expert model is trained by the Super SloMo model through a large number of high-frame-rate videos. (16) (17) (18) The magnitude of the loss function is given by and Decide, The loss value is calculated by the cycle consistency method, which introduces to provide guidance, and introduced Dropout to enhance the generalization ability of model training.

2. The unsupervised video interpolation method based on depth information and random dropout according to claim 1, characterized in that: The process of step 2 is as follows: Step 2.1: Sample the high frame rate training set TRAIN_DATA to generate low frame rate video frames. Sample TRAIN_DATA according to the set sampling value as shown in formula (10). Put the sampled video frames into TRAIN_LOADER. (10) Step 2.2: Group the training data TRAIN_LOADER. Group the TRAIN_LOADER into groups of three adjacent video frames. Discard the redundant video frames. Then, shuffle the order of each group and input them for training. Step 2.3: Randomly cut the TRAIN_LOADER. To reduce training time, cut the video frame into a fixed size (384,384) and save it as a tensor as the final video frame for training. The cut position is randomly selected.

3. The unsupervised video interpolation method based on depth information and random dropout according to claim 2, characterized in that: The process of step three is as follows: Step 3.1: Preprocessing TRAIN_LOADER: Subtract a fixed mean from each set of input video frames as shown in formula (11). After subtracting the mean, in order to enable the model to obtain a larger perception field of view, the input video frames are downsampled by 2 times. (11) in, , , represents the corresponding input video frame, Indicates a fixed value. ; Step 3.2: Put the processed three frames of data into the model training: randomly input the input video frames in the order of T=0, T=1, T=2 or in reverse order, predict the predicted frames at T=t and T=t+1, and then put these two predicted frames into the model to predict the value of T=1 , which is output with Comparing the calculated loss values, the value of t is randomly selected from (0, 1), and its selection range is as shown in formula (12) (12)。 4. The unsupervised video interpolation method based on depth information and random dropout according to claim 1, wherein: The process of step six is ​​as follows: Step 6.1: Extract a set of video frames from the test data TEST_DATA, select the first and last video frames of the dataset from TEST_DATA as the input of the model, and the interval of the video frames is test_num_interp; Step 6.2: Pad the input test data with zeros to determine whether the width and height of the input video frame are multiples of 64. If not, pad the edges with 0 and add them at the end. Step 6.3: Generate prediction frames through the trained model, input two temporally adjacent video frames each time, and output test_num_interp prediction frames.

Images