Real-time video frame insertion method and device based on deep learning model, and product

Abstract

The invention discloses a real-time video frame insertion method, device and product based on a deep learning model, and relates to the field of video processing and transmission. Determining a first local motion area of the first frame image and a second local motion area of the second frame image; the second frame image is the next frame image of the first frame image; inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthetic motion region; and covering the second local motion area with the synthetic motion area to generate a synthetic frame, and the video frame insertion method and device can perform real-time frame insertion on the video under the condition that the GPU performance is not high or even the GPU does not exist.

Description

Technical Field

[0001] The present application relates to the field of video processing and transmission, and in particular to a real-time video interpolation method, device and product based on a deep learning model. Background Art

[0002] With the rapid development of computer technology, software's demand for hardware resources continues to increase. Cloud desktop technology, by offloading complex computing tasks to servers and transmitting desktop data to users over the network for interaction, has been widely used in fields such as enterprise office, education, and healthcare. However, as server frame rates and the number of users increase, hardware and network resources may be in short supply. Users may experience issues such as image freezes and sudden blurring of image quality, which can affect the user experience.

[0003] To solve these problems, video frame interpolation technology can be used to reduce the consumption of hardware and network resources and improve hardware utilization through inter-frame correlation. Therefore, studying desktop data frame insertion processing in cloud desktops has important value and significance.

[0004] Currently, more and more applications can be run directly in the browser. Compared with the traditional client installation method, this method eliminates the necessary step of client installation, greatly reduces the application's dependence on the system, and is receiving more and more attention.

[0005] Existing video frame interpolation techniques primarily rely on deep learning models. For example, the FLAVR model proposed by Kalluri et al. uses three-dimensional spatiotemporal convolution kernels to directly learn motion properties from unlabeled videos, simplifying the frame interpolation model. The EBME model proposed by Jin et al. achieves bidirectional motion estimation by adjusting a series of deformation operations within a pyramid framework. Zhang et al. use attention maps to enhance features and extract motion information, reducing the complexity of inter-frame attention calculation while preserving low-level video image structure information. Lee et al. use variable convolution to expand the model's degrees of freedom, making the video interpolation results more realistic. Teed et al. introduce an attention mechanism into optical flow estimation, iteratively updating and optimizing the optical flow information of adjacent input frames. To address the optical flow reversal problem in traditional optical flow estimation, Huang proposed the RIFE model. This model directly generates intermediate optical flow through end-to-end processing, reducing the error caused by optical flow reversal calculation and achieving real-time frame rate improvement at high resolution. Kong et al. use a pretrained optical flow estimation network to monitor optical flow quality and generate intermediate frames using only an encoder-decoder pair, significantly simplifying the network architecture. He proposed a video interpolation method based on the U-net framework. This method performs real-time interpolation on the client side of video data transmitted from the server, improving the smoothness of the video. The proposed model has a simple structure and can perform real-time interpolation on the Graphics Processing Unit (GPU).

[0006] In summary, existing video interpolation methods for real-time video interpolation rely on mainstream GPU inference. These methods employ complex deep learning models to construct a comprehensive generation model for the corresponding frames. These complex deep learning models require multiple layers of convolutional operations, which slows down the network's inference speed. Furthermore, the comprehensive generation model requires two consecutive frames as input. High video resolutions can hinder real-time frame generation and require a higher GPU configuration. Consequently, existing video interpolation methods may not be able to achieve real-time video interpolation if the remote desktop terminal device has a low-performance GPU or even no GPU at all. Summary of the Invention

[0007] The purpose of this application is to provide a real-time video interpolation method, device, equipment, medium and product based on a deep learning model to solve the problem that real-time video interpolation may not be possible when the GPU performance of the terminal device configured on the remote desktop is not high or even when there is no GPU.

[0008] To achieve the above objectives, this application provides the following solutions.

[0009] In a first aspect, the present application provides a real-time video interpolation method based on a deep learning model, comprising the following steps.

[0010] According to the first frame image and the second frame image of the transmitted video, a first local motion area of ​​the first frame image and a second local motion area of ​​the second frame image are determined; the second frame image is the next frame image of the first frame image.

[0011] The first local motion region and the second local motion region are input into a shallow deep learning video interpolation model, and a synthesized motion region is output.

[0012] The synthetic motion region is covered with the second local motion region to generate a synthetic frame.

[0013] In the second aspect, the present application provides a real-time video interpolation device based on a deep learning model, including the following modules.

[0014] The first local motion area and the second local motion area determination module are used to determine the first local motion area of ​​the first frame image and the second local motion area of ​​the second frame image based on the first frame image and the second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0015] The synthetic motion region output module is used to input the first local motion region and the second local motion region into a shallow deep learning video interpolation model and output a synthetic motion region.

[0016] The synthetic frame generating module is configured to cover the synthetic motion area with the second local motion area to generate a synthetic frame.

[0017] In a third aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements any one of the above-mentioned real-time video interpolation methods based on a deep learning model.

[0018] According to the specific embodiments provided by this application, this application discloses the following technical effects: this application determines the first local motion area and the second local motion area based on the first frame image and the second frame image of the transmitted video, and inputs the first local motion area and the second local motion area into the shallow deep learning video interpolation model, and outputs the synthetic motion area, that is, only the local motion area is input into the shallow deep learning video interpolation model, and the frame generation mode is converted from overall generation to local generation mode, which greatly reduces the input amount of data, improves the inference speed of the model, meets the real-time requirements of the frame generation speed, and the remaining area of ​​the synthetic frame is filled with the original data of the current frame; and compared with the deep learning model, this application adopts a shallow deep learning video interpolation model, avoids multi-layer convolution operations, and further improves the inference speed of the model, thereby realizing real-time interpolation of the video when the GPU performance is not high or even when there is no GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction is given below to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a flowchart of a real-time video frame insertion method based on a deep learning model in one embodiment of the present application.

[0021] Figure 2 Schematic diagram of a shallow deep learning video interpolation model in one embodiment of the present application.

[0022] Figure 3 This is a delay diagram for the original 20fps video.

[0023] Figure 4 This is a delay diagram for a video with a frame rate of 40fps after interpolation. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] The real-time video frame insertion method based on the deep learning model provided in the embodiment of the present application is executed by a computer device, which can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together, such as Figure 1 As shown, in an embodiment of the present application, the method includes the following steps.

[0027] Step 101: Determine a first local motion region of the first frame image and a second local motion region of the second frame image based on a first frame image and a second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0028] Step 102: Input the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and output a synthesized motion region.

[0029] Step 103: Cover the second local motion area with the synthetic motion area to generate a synthetic frame.

[0030] In an exemplary embodiment, step 101 can be replaced by the following steps: decoding the first frame image in a browser and rendering the first frame image; decoding the second frame image in the browser, and subtracting the second frame image from the first frame image, and selecting the pixel with the largest difference as the center point to determine the area of ​​interest; the area of ​​interest is the second local motion area of ​​the second frame image; corresponding the second local motion area to the first frame image to determine the first local motion area.

[0031] In an exemplary embodiment, Figure 2 As shown, step 102 can be replaced by the following steps: the shallow deep learning video interpolation model includes three convolutional layers; the first local motion region and the second local motion region are input into the shallow deep learning video interpolation model to determine a first mask map and a second mask map, and the first local motion region and the second local motion region are warped with the optical flow field to determine a first operation map and a second operation map; the first operation map and the second operation map are linearly weighted summed to determine a synthesized motion region. The three convolutional layers include a convolutional layer with a kernel size of 3×3 and output layer 1, a convolutional layer with a kernel size of 3×3 and output layer 3, and a convolutional layer with a kernel size of 3×3 and output layer 5.

[0032] In an exemplary embodiment, step 102 can be replaced by the following steps, specifically including: detecting whether the browser client has a GPU; if so, using the webGPU method to infer the shallow deep learning video interpolation model to generate a synthetic frame; if not, using the Web-oriented binary format (WebAssembly, Wasm) method to infer the shallow deep learning video interpolation model to generate a synthetic frame.

[0033] In an exemplary embodiment, the synthetic motion area is covered with the second local motion area to generate a synthetic frame, and then the process further includes: rendering the first frame image and the second frame image in a browser, which can double the frame rate.

[0034] This application also provides an application scenario to illustrate the technical solution of this application.

[0035] First, render the first frame of the image transmitted in the browser and save it as frame1.

[0036] Next, we perform a subtraction between the second frame (frame 2) and the first frame (frame 1). We then select the pixel with the largest difference as the center and define a 200px × 200px square region of interest as the second local motion region. The default value of 200px × 200px can be adjusted based on actual needs.

[0037] Then, the second local motion area motion_area2 of frame2 and the first local motion area motion_area1 corresponding to frame1 are fed into the trained shallow deep learning video interpolation model, and the loss function of the shallow deep learning video interpolation model is set to L1-loss.

[0038] The training dataset consists of a conventional video interpolation dataset and a self-made dataset. It should be noted that the self-made dataset is generally a video of various software operations. For example, if you need to interpolate an Adobe Photoshop video, you can use the Adobe Photoshop operation video to create a training dataset; if you need to interpolate a CAD video, you can use the CAD operation video to create a training dataset.

[0039] Finally, a 200×200 synthetic motion area is output, and the pixel values ​​of the synthetic motion area are used to overwrite the second local motion area motion_area2 corresponding to frame2 to obtain a new synthetic frame, thereby achieving the purpose of only updating the motion area. The frame updated by the model is then displayed, thereby displaying the latest transmitted frame.

[0040] Furthermore, by detecting whether the client has a GPU, we can choose to use Wasm or webGPU to infer the shallow deep learning video interpolation model. If there is no GPU, the Wasm method is used; if there is a GPU, the webGPU method is used.

[0041] The real-time video interpolation method based on deep learning model provided by this application was tested in Chrome browser. The entire test process was tested on a device with CPU I9-13900K and graphics card NVIDIA GeForce RTX 4090. The test video resolution was 1080P. In order to simplify the implementation and ensure the universality of the test results, in this experiment, the wasm method was used to interpolate the motion area. The experimental results are as follows: Figure 3 and Figure 4 As shown, from Figure 3 and Figure 4 The latency at 20fps is 68ms, and at 40fps it is 84ms, an increase of 16ms. This increase is mainly due to the conversion between YUV420p and RGB, and model inference. It can be seen that these operations have little impact on latency. This application, by only inserting frames in the moving areas of the video, ultimately increases the frame rate from 20fps to 40fps, while maintaining a non-significant increase in latency when playing the video in the browser, thus improving video viewing smoothness.

[0042] Based on the same inventive concept, the present application also provides a real-time video interpolation device based on a deep learning model for implementing the real-time video interpolation method based on a deep learning model. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more embodiments of the real-time video interpolation device based on a deep learning model provided below can be found in the limitations of the real-time video interpolation method based on a deep learning model above, and will not be repeated here.

[0043] In an exemplary embodiment, a real-time video frame insertion device based on a deep learning model is provided, which includes the following modules.

[0044] The first local motion area and the second local motion area determination module are used to determine the first local motion area of ​​the first frame image and the second local motion area of ​​the second frame image based on the first frame image and the second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0045] The synthetic motion region output module is used to input the first local motion region and the second local motion region into a shallow deep learning video interpolation model and output a synthetic motion region.

[0046] The synthetic frame generating module is configured to cover the synthetic motion area with the second local motion area to generate a synthetic frame.

[0047] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0048] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0049] The processor involved in each embodiment provided in this application may be a central processing unit or a graphics processing unit, but is not limited thereto.

[0050] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A real-time video frame insertion method based on a deep learning model, characterized in that: The real-time video frame insertion method comprises: Determine, according to the first frame image and the second frame image of the transmitted video, a first local motion region of the first frame image and a second local motion region of the second frame image; the second frame image is a frame image next to the first frame image; Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthesized motion region; The synthetic motion region is covered with the second local motion region to generate a synthetic frame.

2. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Determining a first local motion region of the first frame image and a second local motion region of the second frame image according to a first frame image and a second frame image of the transmitted video specifically includes: Decoding the first frame of image in a browser, and rendering the first frame of image; Decoding the second frame image in the browser, performing a difference between the second frame image and the first frame image, and selecting a pixel point with the largest difference as a center point to determine an area of ​​interest; the area of ​​interest is a second local motion area of ​​the second frame image; The second local motion region is matched to the first frame image to determine a first local motion region.

3. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthesized motion region, specifically comprising: the shallow deep learning video interpolation model comprises three convolutional layers; Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model to determine a first mask map and a second mask map, and performing warp operations on the first local motion region and the second local motion region and the optical flow field respectively to determine a first operation map and a second operation map; The first operation map and the second operation map are linearly weighted summed to determine a synthetic motion area.

4. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Covering the second local motion area with the synthetic motion area to generate a synthetic frame specifically includes: Detect whether a GPU exists on the client of the browser, and obtain a first determination result; If the first judgment result is that the client of the browser has a GPU, the shallow deep learning video interpolation model is inferred by using a webGPU method, and the synthetic motion region is interpolated into the second local motion region to generate a synthetic frame; If the first judgment result is that there is no GPU on the client of the browser, the shallow deep learning video interpolation model is inferred using Wasm, and the synthetic motion area is interpolated into the second local motion area to generate a synthetic frame.

5. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Covering the second local motion area with the synthetic motion area to generate a synthetic frame, and then further comprising: The first frame image and the second frame image are rendered in a browser.

6. A real-time video frame insertion device based on a deep learning model, characterized in that: The real-time video frame insertion device based on the deep learning model includes: A first local motion region and a second local motion region determining module, used to determine a first local motion region of the first frame image and a second local motion region of the second frame image according to a first frame image and a second frame image of the transmitted video; the second frame image is a frame image next to the first frame image; A synthetic motion region output module, used for inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthetic motion region; The synthetic frame generation module is used to cover the second local motion area with the synthetic motion area to generate a synthetic frame.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the real-time video interpolation method based on a deep learning model described in any one of claims 1 to 5 is implemented.

Images