Method and apparatus for generating video intermediate frames

Abstract

The application discloses a method and device for generating an intermediate frame of a video, wherein the method comprises: performing a warping operation on two target video frames based on a bidirectional optical flow between the two target video frames to obtain two pictures; predicting a network depth of a frame synthesis network matched by each picture sub-region according to a similarity of two sub-pictures corresponding to the picture sub-region in the two pictures, and adopting a strategy that a network depth is larger as a similarity is smaller; for each picture sub-region, performing a synthesis processing on a corresponding sub-picture according to a network depth matched by the picture sub-region by using the frame synthesis network; and splicing all images obtained by the synthesis processing according to the picture sub-regions to obtain an intermediate frame of the target video frame. By using the application, the operation cost can be effectively reduced, and the generation efficiency and quality of the intermediate frame can be improved.

Description

Technical Field

[0001] This invention relates to computer vision technology, and in particular to a method and apparatus for generating intermediate frames in a video. Background Technology

[0002] Video frame interpolation is an important application in computer vision. It aims to increase the frame rate of a video by synthesizing frames that are not present between consecutive frames, making motion smoother and improving the viewer's experience. For example, some older videos, limited by the equipment used at the time, typically only have 25 or 30 FPS, while modern high-end televisions usually support 120 FPS. Playing low frame rate videos on a modern television results in a choppy experience and fails to utilize the high performance of the television. To solve this problem, artificial intelligence technology can be used to increase the video frame rate to 120 FPS through frame interpolation. Furthermore, video frame interpolation technology has wide applications in video compression, perspective synthesis, and adaptive streaming media. In particular, with the growing popularity of the metaverse concept, real-time high-definition video interaction is becoming increasingly important. However, real-time transmission of high-definition, high-frame-rate video streams puts significant strain on network bandwidth. Therefore, under limited bandwidth conditions, transmitting lower frame rate video streams and then using video frame interpolation on the client side to convert the low frame rate video to a high frame rate video is a highly promising solution.

[0003] Currently, deep learning algorithms have achieved significant results in the field of video frame interpolation. In particular, optical flow-based pixel synthesis is the mainstream algorithm framework in this field. This algorithm mainly includes two steps: 1) estimating the optical flow between the input frame and the target frame; 2) using optical flow to guide the synthesis of intermediate frames. Optical flow describes pixel-level motion between consecutive frames. Through optical flow-based warp operations (forward-warping or backward-warping), pixels in the input frame can be moved to the intermediate frame. However, warp operations can introduce artifacts. Especially in cases of occlusion, forward-warping can produce black holes, and backward-warping can produce duplicate objects. Therefore, after warp operations, a frame synthesis network is typically needed to fuse the pixel and feature information from the warp operation to generate the final intermediate frame, eliminating the artifacts after warp and inferring the appearance of occluded objects. Generally speaking, a frame synthesis network is an encoder-decoder structure, also known as the U-Net structure.

[0004] In the process of developing this invention, the inventors discovered that existing schemes for generating intermediate frames based on Warp operation results suffer from high computational overhead and cannot guarantee the image quality of intermediate frames. Through careful research and analysis, the inventors found the specific reasons for the above problems to be as follows:

[0005] In existing solutions, when generating intermediate frames using a frame synthesis network, the same network depth (i.e., the number of stages in the frame synthesis network) is used for synthesis of all pixels in the image. This means that the computational load allocated to each pixel of the input image is the same during intermediate frame generation. However, in practical applications, the synthesis difficulty varies for different regions of the intermediate frame. Regions with higher optical flow intensity or more complex (non-linear motion) are more difficult to synthesize than regions with lower optical flow intensity (nearly static) or simpler (linear motion) regions. Therefore, because existing solutions use a synthesis network of the same depth to process all image regions, the synthesis depth used in some regions may not match the corresponding synthesis difficulty. For example, for some regions with lower optical flow intensity (or simpler regions), a shallower frame synthesis network should be used, but a deeper frame synthesis network is used instead, resulting in a waste of computational resources. Furthermore, for super-resolution tasks, some regions processed using shallow networks may even yield better results than those processed using deep networks. Therefore, from a visual perspective, using the same network depth for frame synthesis calculations across all areas of the image often cannot guarantee optimal image quality. Summary of the Invention

[0006] In view of this, the main objective of the present invention is to provide a method and apparatus for generating intermediate frames of video, which can reduce computational overhead and improve the quality of intermediate frames.

[0007] To achieve the above objectives, the technical solution proposed in this embodiment of the invention is as follows:

[0008] A method for generating intermediate frames in a video, comprising:

[0009] Based on the bidirectional optical flow between two target video frames, the target video frames are warped to obtain two images;

[0010] Based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, a strategy of increasing network depth with lower similarity is adopted to predict the network depth of the frame synthesis network matching the corresponding sub-region of the image.

[0011] For each of the aforementioned sub-regions of the image, the corresponding sub-image is synthesized using a frame synthesis network according to the network depth it matches;

[0012] According to the sub-regions of the image, all the images obtained by the synthesis process are stitched together to obtain the intermediate frame of the target video frame.

[0013] This invention also proposes a device for generating intermediate video frames, comprising:

[0014] The video frame processing unit is used to perform a warping operation on the target video frames based on the bidirectional optical flow between the two target video frames to obtain two images;

[0015] The network depth matching unit is used to predict the network depth of the frame synthesis network matching the corresponding sub-region of the image based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, and adopts the strategy that the smaller the similarity, the greater the network depth.

[0016] A local compositing unit is used to perform compositing processing on the corresponding sub-image for each of the image sub-regions using a frame compositing network according to the network depth it matches;

[0017] The stitching unit is used to stitch together all the images obtained from the synthesis process according to the sub-regions of the screen to obtain the intermediate frame of the target video frame.

[0018] This invention also proposes a device for generating intermediate video frames, including a processor and a memory;

[0019] The memory stores an application program that can be executed by the processor, which causes the processor to execute the video intermediate frame generation method described above.

[0020] This invention also proposes a computer-readable storage medium storing computer-readable instructions for performing the video intermediate frame generation method described above.

[0021] This invention also proposes a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the video intermediate frame generation method described above.

[0022] In summary, in the video intermediate frame generation method and apparatus proposed in this embodiment of the invention, it is necessary to compare the image alignment degree between target video frame pairs after the distortion operation. According to the image alignment degree (i.e., similarity) of each local image region (i.e., sub-region of the image) in the picture, the network depth of the matching frame synthesis network is predicted for each local image region of the image. In this way, when processing with the frame synthesis network, each local image region of the image will adopt the optimal network depth, rather than all image regions adopting the same network depth. This ensures that each local image region adopts computing resources that match its own synthesis difficulty, thereby effectively reducing the overall computing load during frame synthesis processing, avoiding waste of computing resources, and improving synthesis efficiency. On the other hand, it ensures the frame synthesis effect corresponding to each local image region, thereby improving the image quality of the intermediate frame.

[0023] Furthermore, the above scheme requires prediction of the network depth for frame synthesis based on the image after the warping operation. Thus, by first aligning the target video frame pairs using the warping operation, even if the spatial positions of corresponding pixels do not completely overlap, they will not deviate too far. This allows the size of the local image region to be set relatively small. On the one hand, this can overcome the strong constraint of optical flow scale on the size of the local region, enabling accurate estimation of the corresponding intermediate frame image even in the case of large-scale optical flow, thus enhancing robustness in the case of large-scale optical flow. On the other hand, for low-resolution videos, it can also dynamically determine the network depth of each local image region and perform corresponding local image synthesis processing. Therefore, the embodiments of the present invention are applicable not only to high-resolution videos but also to low-resolution videos, effectively expanding the application scope. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0025] Figure 2 This is an example diagram illustrating the prediction of the matching frame synthesis network depth for each sub-region of the image in an embodiment of the present invention.

[0026] Figure 3 This is a comparative illustration of the effects of the embodiments of the present invention and existing video frame interpolation schemes in a single-frame video interpolation scenario.

[0027] Figure 4 This is an example diagram illustrating the effect of generating multiple intermediate frames using an embodiment of the present invention.

[0028] Figure 5 This is a schematic diagram of the device structure according to an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0030] Figure 1 This is a schematic diagram of the method flow of an embodiment of the present invention, such as... Figure 1 As shown, the method for generating intermediate video frames implemented in this embodiment mainly includes the following steps:

[0031] Step 101: Based on the bidirectional optical flow between the two target video frames, perform a warping operation on the target video frames to obtain two images.

[0032] It should be noted that, in this embodiment, to reduce frame synthesis computational overhead and improve the generation quality of intermediate frames, each local image region needs to be processed separately in subsequent steps. Considering the significant pixel shift caused by large-scale optical flow, if the scale of the optical flow exceeds the size of the local image region, theoretically, it will be impossible to estimate the intermediate frame well. Therefore, the size of the local image region needs to be set relatively large so that the frame synthesis network can handle the pixel shift caused by large-scale optical flow well within the local image region. However, if the local image region is too large, the frame synthesis network can only perform block processing on high-resolution video, and cannot perform adaptive block processing on low-resolution video, thus limiting the solution to high-resolution video.

[0033] To address the aforementioned application limitations, this embodiment of the invention requires a warping operation on the two target video frames before frame synthesis. This is because, based on a forward-warping operation from the input video frame to the intermediate frame, or a backward-warping operation from the intermediate frame to the input video frame, the pixels of the two input target video frames can be aligned to the intermediate frame. Therefore, by performing the warping operation first, the positional deviation between corresponding pixels in the two target video frames can be reduced, ensuring that even if the spatial positions of corresponding pixels do not completely overlap, they will not be significantly misaligned. This eliminates the need to set the local image regions to be too large when performing separate synthesis processing on local image regions in subsequent steps. Thus, on the one hand, the warping operation can eliminate the strong correlation between the optical flow scale and the size of the local image region in large-scale optical flow scenarios, eliminating the need to set the local image region to be very large in order to accurately estimate the intermediate frame, thereby enhancing the robustness of the solution in large-scale optical flow scenarios; on the other hand, for low-resolution videos, it is also possible to dynamically determine the network depth of each local image region locally and perform corresponding synthesis processing, so that the embodiments of the present invention are applicable not only to high-resolution videos but also to low-resolution videos, thereby effectively expanding the application scope of the solution.

[0034] In addition, since the convolutional neural network (CNN) features after the warping operation can provide rich contextual information, the frame synthesis network can also use these CNN features as input while taking the warped image as input during subsequent frame synthesis processing, thereby improving the robustness of intermediate frame generation.

[0035] In practical applications, existing optical flow estimation methods can be used to obtain the bidirectional optical flow between two target video frames, which will not be elaborated here.

[0036] Considering that forward-warping operations are easier to implement than backward-warping operations, in order to further reduce the computational overhead of the embodiments of the present invention and improve the generation efficiency of intermediate frames, in one embodiment, the warping operation in step 101 can be specifically implemented as a forward-warping operation.

[0037] Step 102: Based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, and using the strategy that the smaller the similarity, the greater the network depth, predict the network depth of the frame synthesis network matching the corresponding sub-region of the image.

[0038] This step is used to determine the matching frame synthesis network depth for each local image region (i.e., sub-region of the image). By predicting the matching network depth for each local image region separately, the waste of computational resources during frame synthesis can be avoided, the overall frame synthesis overhead can be reduced, and the efficiency and quality of generating intermediate frames can be improved.

[0039] It should be noted that the network depth in this application specifically refers to the number of stages in the frame synthesis network. In a convolutional neural network, a stage is associated with the size of the output feature map. Within each stage, the feature map size is essentially the same. If the feature map size changes significantly, it will move to the next stage. That is, if multiple consecutive convolutional layers have the same output resolution, these convolutional layers are in the same stage; the initial stage of a convolutional layer is 1; when a convolutional layer performs a downsampling operation, the resolution of its output feature map is reduced, and the stage number increases by 1.

[0040] Figure 2 An example is given, which uses two images after a warp operation to predict the matching frame synthesis network depth for each sub-region of the image. Figure 2 The second row is a magnified view of a portion of the sub-images in the left-hand image of the first row. For example... Figure 2As shown, when the images of the two input video frames obtained from the warp operation are superimposed (as shown in the left image of the first row), it can be seen that the pixels in most areas overlap well (as shown in the lower right sub-image of the left image of the first row, and the leftmost image of the second row is a magnified view of the lower right sub-image). Both warped images can be regarded as coarse predictions of the intermediate frame. If the consistency of the two predicted images in a certain local area is very high (pixels are perfectly aligned), the similarity between the warped local image patch and the local image patch of the intermediate frame will also be very high. For such local image patches, a shallow frame synthesis network can obtain high-quality image synthesis results. However, the inventors found that due to inaccurate optical flow estimation or occlusion between objects, there are also some local image patches (such as the two sub-images in the middle of the first row in the left image of the first row, and the two magnified views of these two sub-images in the right image of the second row) where many pixels are not aligned. In order to accurately infer the local image patch corresponding to the intermediate frame from these inconsistent local image patches, the frame synthesis network often needs to have a large model space and receptive field. Therefore, a deeper frame synthesis network is needed when processing these local blocks. Thus, in step 102, by performing a matching network depth prediction for each local image region based on the similarity between the two sub-images corresponding to that region in the two input frames, and adopting a strategy of increasing network depth for smaller similarities, the optimal depth of the frame synthesis network used for each local image region can be effectively ensured.

[0041] Specifically, in one implementation, a fully convolutional neural network (FCN) can be used to predict the network depth of the frame synthesis network matching each sub-region of the image based on the two images obtained from the above-mentioned warping operation.

[0042] In practical applications, the optical flow estimation network, the fully convolutional neural network, and the frame synthesis network can be integrated into an intermediate frame generation model in advance and trained together. During training, two sample video frames are used as the input data of the model to generate the corresponding intermediate frames. Then, the intermediate frames obtained based on the output results and the corresponding real intermediate frames are used to calculate the loss function. Finally, the network parameters of the three networks are optimized and adjusted based on the obtained loss function value.

[0043] The size of the sub-region of the image is used to divide the image into several local image blocks (i.e., sub-images) so as to dynamically and locally determine the depth of the frame synthesis network. Specifically, the size can be set in advance by those skilled in the art according to actual needs. As analyzed above, in the embodiments of the present invention, since a distortion operation is used on the input video frame, the size of the sub-region of the image can be set to be relatively small. Preferably, in one embodiment, the size of the sub-region of the image can be no greater than 64×64dpi.

[0044] Step 103: For each of the image sub-regions, use the frame synthesis network to perform synthesis processing on the corresponding sub-image according to the network depth it matches.

[0045] In this step, based on the network depth prediction results of each sub-region of the image obtained in step 102, frame synthesis networks of different depths are used to process the synthesis of different image patches (i.e., sub-images). For sub-regions of the image with poor alignment after the warp operation, the synthesis is more difficult and requires a deeper network; for image patches of sub-regions of the image with lower synthesis difficulty, a shallower network can be used for good synthesis. It is worth noting that in this embodiment of the invention, multiple different frame synthesis networks are not required. Only one relatively deep frame synthesis network is needed. During frame synthesis, for each sub-region of the image, the parameters of the corresponding depth of the frame synthesis network are called according to the network depth corresponding to that sub-region. Since in practical applications, for two consecutive high-resolution images, the regions with truly complex or large-scale optical flow are often small, while most regions either have no optical flow or only simple linear optical flow. Thus, most sub-regions of the image only need to be processed using a shallower frame synthesis network. Therefore, in this step, differentiating different sub-regions of the image and using matching network depth parameters for synthesis processing can usually greatly reduce the amount of computation and improve the efficiency of frame synthesis processing. In addition, since the network depth of each sub-region of the image is matched with the similarity of its two corresponding sub-images, the influence of using network depth parameters that are not matched with the synthesis difficulty on the synthesis results can be avoided, effectively ensuring the generation quality of intermediate frames.

[0046] Considering that the existing conventional U-Net network structure is relatively simple and easy to apply, in one embodiment, the frame synthesis network can be specifically implemented as a U-Net network.

[0047] Step 104: According to the sub-regions of the screen, stitch together all the images obtained by the synthesis process to obtain the intermediate frame of the target video frame.

[0048] In this step, according to the sub-region of the screen to which each image belongs, and based on the positional relationship between the sub-regions, all the images obtained in step 103 are stitched together to obtain a complete video frame, which is the intermediate frame between two target video frames.

[0049] As can be seen from the above technical solution, in this embodiment of the invention, based on the images of two target video frames after the distortion operation, different local image regions are distinguished, and frame synthesis network depths that match the synthesis difficulty are used for synthesis processing separately. This significantly reduces computational overhead and improves the efficiency and quality of generating intermediate frames. Furthermore, by performing a distortion operation on the target video first, the solution can better handle large-scale optical flow scenarios and is applicable to both high-resolution and low-resolution videos. The specific applications of this embodiment of the invention are further illustrated below with several application examples.

[0050] 1. Single-frame video interpolation

[0051] A common video frame interpolation requirement is to input two video frames, frame-0 and frame-1, and synthesize the middle frame, frame-0.5, through a model. Figure 3 Example diagrams comparing the effects of embodiments of the present invention and existing video frame interpolation algorithms are provided. Figure 3 As shown in the figure, the first column represents two overlapping input video frames, the second column is a reference to the actual intermediate frames, columns three through six are intermediate frames generated using four of the most advanced frame interpolation methods (CAIN, ABME, VFIformer, and IFRNetlarge), and the seventh column is an intermediate frame generated using the embodiment of this invention. The second and fourth rows are magnified views of the corresponding columns in the first and third rows, respectively, making it easier to see the subtle differences between the different images. The magnified views demonstrate that the embodiment of this invention achieves good results regardless of whether the optical flow is extremely large (second row) or the localized complex nonlinear optical flow (fourth row).

[0052] 2. Multi-frame video interpolation

[0053] When the original video has a low frame rate (e.g., 25 FPS), it is often necessary to insert multiple frames between two consecutive frames to achieve a higher frame rate (e.g., 120 FPS). Based on a linear weighting method, the optical flow from the input frame to the intermediate frame at any time can be approximated from the bidirectional optical flow. Then, by combining forward-warping operations, the intermediate frame at any time can be synthesized. Figure 4 Example diagrams showing the effects of generating multiple intermediate frames using embodiments of the present invention are provided. Figure 4 The first row on the right is a reference to actual intermediate frames, and the second row on the right is an example of multiple intermediate frames obtained using embodiments of the present invention, such as... Figure 4 As shown, the intermediate frames obtained using the embodiments of the present invention are very close to the actual video frames.

[0054] 3. Other applications based on video frame interpolation

[0055] Because the embodiments of this invention can efficiently and robustly estimate intermediate frames between adjacent video frames, it can also be applied to many downstream applications based on video frame interpolation algorithms, including video compression, video enhancement, and viewpoint synthesis. In particular, within the concept of the metaverse, the interaction of real-time high-definition video content is becoming increasingly important. Under limited bandwidth conditions, transmitting low-frame-rate video streams and using video frame interpolation technology on the client side to convert low-frame-rate video into high-frame-rate video will see widespread application.

[0056] Based on the above method embodiments, this invention also proposes a device for generating intermediate video frames, such as... Figure 5 As shown, the device includes:

[0057] The video frame processing unit is used to perform a warping operation on the target video frames based on the bidirectional optical flow between the two target video frames to obtain two images;

[0058] The network depth matching unit is used to predict the network depth of the frame synthesis network matching the corresponding sub-region of the image based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, and adopts the strategy that the smaller the similarity, the greater the network depth.

[0059] A local compositing unit is used to perform compositing processing on the corresponding sub-image for each of the image sub-regions using a frame compositing network according to the network depth it matches;

[0060] The stitching unit is used to stitch together all the images obtained from the synthesis process according to the sub-regions of the screen to obtain the intermediate frame of the target video frame.

[0061] It should be noted that the above methods and apparatus are based on the same inventive concept. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0062] Based on the above method embodiments, this invention also proposes a video intermediate frame generation device, including a processor and a memory; the memory stores an application program executable by the processor, which causes the processor to execute the video intermediate frame generation method described above. Specifically, a system or apparatus equipped with a storage medium can be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus can read and execute the program code stored in the storage medium. Furthermore, the operating system or other devices operating on the computer can perform some or all of the actual operations through instructions based on the program code. The program code read from the storage medium can also be written to a memory located in an expansion board inserted into the computer or to a memory located in an expansion unit connected to the computer, and then, based on the instructions of the program code, a CPU or other device installed on the expansion board or expansion unit can execute some or all of the actual operations, thereby realizing the functions of any of the above embodiments of the video intermediate frame generation method.

[0063] Specifically, the memory can be implemented as various storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory, and programmable programmable read-only memory (PROM). The processor can be implemented as one or more central processing units (CPUs) or one or more field-programmable gate arrays (FPGAs), wherein the FPGA integrates one or more CPU cores. Specifically, the CPU or CPU core can be implemented as a CPU or an MCU.

[0064] This application also implements a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the video intermediate frame generation method described above.

[0065] It should be noted that not all steps and modules in the above processes and structural diagrams are mandatory; some steps or modules can be omitted as needed. The execution order of the steps is not fixed and can be adjusted as required. The division of modules is merely for the convenience of description and functional division. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices.

[0066] The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuitry or logic devices (such as dedicated processors, such as FPGAs or ASICs) to perform specific operations. A hardware module may also include programmable logic devices or circuitry (such as general-purpose processors or other programmable processors) temporarily configured by software to perform specific operations. The choice between mechanical implementation, dedicated permanent circuitry, or temporarily configured circuitry (such as software-configured circuitry) can be made based on cost and time considerations.

[0067] In this document, "illustrative" means "serving as an example, illustration, or description," and any illustration or embodiment described herein as "illustrative" should not be construed as a preferred or more advantageous technical solution. For the sake of brevity, the figures only schematically represent the parts relevant to the invention and do not represent their actual structure as a product. Furthermore, for the sake of clarity and ease of understanding, in some figures, components with the same structure or function are only schematically depicted, or only one is labeled. In this document, "a" does not mean that the number of relevant parts of the invention is limited to "only one," and "a" does not exclude the possibility that the number of relevant parts of the invention is "more than one." In this document, terms such as "upper," "lower," "front," "rear," "left," "right," "inner," and "outer" are used only to indicate the relative positional relationship between relevant parts, and not to limit the absolute position of these relevant parts.

[0068] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0069] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating intermediate frames in a video, characterized in that, include: Based on the bidirectional optical flow between two target video frames, the target video frames are warped to obtain two images; Based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, a strategy of increasing network depth with lower similarity is adopted to predict the network depth of the frame synthesis network matching the corresponding sub-region of the image. For each of the aforementioned sub-regions of the image, the corresponding sub-image is synthesized using a frame synthesis network according to the network depth it matches; According to the sub-regions of the image, all the images obtained by the synthesis process are stitched together to obtain the intermediate frame of the target video frame.

2. The method according to claim 1, characterized in that, The warping operation is a forward-warping operation.

3. The method according to claim 1, characterized in that, The size of the sub-region of the image is no greater than 64×64dpi.

4. The method according to claim 1, characterized in that, Using a fully convolutional neural network (FCN), the network depth is predicted based on the two images.

5. The method according to claim 1, characterized in that, The frame synthesis network is a U-Net network.

6. A device for generating intermediate frames in a video, characterized in that, include: The video frame processing unit is used to perform a warping operation on the target video frames based on the bidirectional optical flow between the two target video frames to obtain two images; The network depth matching unit is used to predict the network depth of the frame synthesis network matching the corresponding sub-region of the image based on the similarity between the two sub-images corresponding to each sub-region of the image in the two images, and adopts the strategy that the smaller the similarity, the greater the network depth. A local compositing unit is used to perform compositing processing on the corresponding sub-image for each of the image sub-regions using a frame compositing network according to the network depth it matches; The stitching unit is used to stitch together all the images obtained from the synthesis process according to the sub-regions of the screen to obtain the intermediate frame of the target video frame.

7. A device for generating intermediate frames in a video, characterized in that, Including processor and memory; The memory stores an application program that can be executed by the processor, which causes the processor to perform the method for generating intermediate video frames as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It contains computer-readable instructions for performing the method for generating intermediate video frames as described in any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the method for generating intermediate video frames as described in any one of claims 1 to 5.

Images