A real-time video frame insertion method, device and product based on deep learning model

Abstract

The present application discloses a real-time video interpolation method, device and product based on a deep learning model, which relates to the field of video processing and transmission. The method comprises determining a first local motion region of a first frame image and a second local motion region of a second frame image according to a first frame image and a second frame image of a transmitted video; the second frame image is the next frame image of the first frame image; the first local motion region and the second local motion region are input into a shallow deep learning video interpolation model, and a synthetic motion region is output; the synthetic motion region is covered with the second local motion region to generate a synthetic frame. The present application can perform real-time interpolation of a video even when the GPU performance is not high or even when there is no GPU.

Description

Technical Field

[0001] The present application relates to the field of video processing and transmission, and in particular to a real-time video interpolation method, device and product based on a deep learning model. Background Art

[0002] With the rapid development of computer technology, the demand for hardware resources by software is increasing. Cloud desktop technology has been widely used in the fields of corporate office, education, and medical care by transferring complex computing tasks to the server and transmitting desktop data to the user end through the network for interaction. However, with the increase in server frame rate and the number of users, hardware and network resources may be in short supply. Users may encounter problems such as image freezes and sudden blurring of image quality during use, which affects the user experience.

[0003] In order to solve these problems, video frame interpolation technology can be used to reduce the consumption of hardware and network resources and improve hardware utilization through inter-frame correlation. Therefore, it is of great value and significance to study the desktop data frame insertion processing in cloud desktop.

[0004] Currently, more and more applications can be run directly in the browser. Compared with the traditional client installation method, this method eliminates the necessary step of client installation, greatly reduces the application's dependence on the system, and is receiving more and more attention.

[0005] Existing video frame interpolation technologies mainly rely on deep learning models. For example, the FLAVR model proposed by Kalluri et al. uses a three-dimensional spatiotemporal convolution kernel to directly learn motion properties from unlabeled videos, simplifying the frame interpolation model. The EBME model proposed by Jin et al. achieves bidirectional motion estimation by adjusting a series of deformation operations in a pyramid framework. Zhang et al. use attention maps to enhance features and extract motion information, reducing the complexity of inter-frame attention calculation while retaining low-level video image structure information. The variable convolution used by Lee et al. expands the degree of freedom of the model and makes the video interpolation results more realistic. Teed et al. introduced the attention mechanism into optical flow estimation to iteratively update and optimize the optical flow information of adjacent input frames. In response to the optical flow reversal problem in traditional optical flow estimation, Huang proposed the RIFE model, which directly generates intermediate optical flow through end-to-end, reduces the error caused by optical flow reversal calculation, and achieves real-time frame rate improvement at high resolution. Kong et al. use a pre-trained optical flow estimation network to monitor the optical flow quality and only use a pair of encoder-decoders to generate intermediate frames, greatly simplifying the network architecture. He proposed a video interpolation method based on the U-net framework. This method performs real-time interpolation on the client side for video data transmitted from the server, improving the smoothness of the video. The proposed model has the characteristics of simple structure and can perform real-time interpolation on the graphics processing unit (GPU).

[0006] In summary, existing video interpolation methods for real-time video interpolation are all based on mainstream GPU reasoning, that is, using complex deep learning models to build the overall generation model of the corresponding frames. Complex deep learning models require multi-layer convolution operations, which reduces the reasoning speed of the network, and the overall generation model of the frame requires two consecutive frames as the input of the model. If the video resolution is too high, the real-time frame generation cannot be met, and a higher GPU configuration is required. Therefore, the existing video interpolation methods may not be able to perform real-time video interpolation when the GPU performance of the terminal device of the remote desktop is not high, or even when there is no GPU. Summary of the invention

[0007] The purpose of this application is to provide a real-time video interpolation method, device, equipment, medium and product based on a deep learning model to solve the problem that real-time video interpolation may not be possible when the GPU performance of the terminal device configured on the remote desktop is not high or even when there is no GPU.

[0008] To achieve the above objectives, this application provides the following solutions.

[0009] In a first aspect, the present application provides a real-time video interpolation method based on a deep learning model, comprising the following steps.

[0010] According to the first frame image and the second frame image of the transmitted video, a first local motion area of ​​the first frame image and a second local motion area of ​​the second frame image are determined; the second frame image is the next frame image of the first frame image.

[0011] The first local motion region and the second local motion region are input into a shallow deep learning video interpolation model, and a synthetic motion region is output.

[0012] The synthetic motion region is covered with the second local motion region to generate a synthetic frame.

[0013] In a second aspect, the present application provides a real-time video interpolation device based on a deep learning model, comprising the following modules.

[0014] The first local motion area and the second local motion area determination module are used to determine the first local motion area of ​​the first frame image and the second local motion area of ​​the second frame image according to the first frame image and the second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0015] The synthetic motion region output module is used to input the first local motion region and the second local motion region into a shallow deep learning video interpolation model and output a synthetic motion region.

[0016] The synthetic frame generation module is used to cover the second local motion area with the synthetic motion area to generate a synthetic frame.

[0017] In a third aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned real-time video interpolation methods based on a deep learning model.

[0018] According to the specific embodiments provided by the present application, the present application discloses the following technical effects: the present application determines a first local motion area and a second local motion area according to a first frame image and a second frame image of a transmitted video, and inputs the first local motion area and the second local motion area into a shallow deep learning video interpolation model, and outputs a synthetic motion area, that is, only inputs the local motion area into the shallow deep learning video interpolation model, and converts the frame generation mode from overall generation to local generation mode, which greatly reduces the amount of data input, improves the inference speed of the model, meets the real-time requirements of the frame generation speed, and fills the remaining area of ​​the synthetic frame with the original data of the current frame; and compared with the deep learning model, the present application adopts a shallow deep learning video interpolation model, avoids multi-layer convolution operations, and further improves the inference speed of the model, thereby realizing real-time interpolation of the video when the GPU performance is not high or even when there is no GPU. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a flowchart of a real-time video frame insertion method based on a deep learning model in one embodiment of the present application.

[0021] Figure 2 Schematic diagram of a shallow deep learning video interpolation model in one embodiment of the present application.

[0022] Figure 3 This is a delay diagram for the original 20fps video.

[0023] Figure 4 This is a delay diagram of a video with a frame rate of 40fps after frame insertion. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0026] The real-time video frame insertion method based on the deep learning model provided in the embodiment of the present application is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together, such as Figure 1 As shown, in an embodiment of the present application, the method includes the following steps.

[0027] Step 101: Determine a first local motion region of the first frame image and a second local motion region of the second frame image according to a first frame image and a second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0028] Step 102: Input the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and output a synthesized motion region.

[0029] Step 103: Cover the second local motion area with the synthetic motion area to generate a synthetic frame.

[0030] In an exemplary embodiment, step 101 can be replaced by the following steps: decoding the first frame image in a browser and rendering the first frame image; decoding the second frame image in the browser, subtracting the second frame image from the first frame image, and selecting the pixel with the largest difference as the center point to determine the area of ​​interest; the area of ​​interest is the second local motion area of ​​the second frame image; corresponding the second local motion area to the first frame image to determine the first local motion area.

[0031] In an exemplary embodiment, Figure 2 As shown, step 102 can be replaced by the following steps: the shallow deep learning video interpolation model includes three convolutional layers; the first local motion region and the second local motion region are input into the shallow deep learning video interpolation model to determine the first mask map and the second mask map, and the first local motion region and the second local motion region are respectively subjected to warp operation with the optical flow field to determine the first operation map and the second operation map; the first operation map and the second operation map are subjected to linear weighted summation processing to determine the synthetic motion region. Among them, the three convolutional layers include a convolutional layer with a kernel size of 3×3 and an output layer 1, a convolutional layer with a kernel size of 3×3 and an output layer 3, and a convolutional layer with a kernel size of 3×3 and an output layer 5.

[0032] In an exemplary embodiment, step 102 can be replaced by the following steps, specifically including: detecting whether there is a GPU on the client of the browser, and if so, using the webGPU method to infer the shallow deep learning video interpolation model to generate a synthetic frame; if not, using a Web-oriented binary format (WebAssembly, Wasm) method to infer the shallow deep learning video interpolation model to generate a synthetic frame.

[0033] In an exemplary embodiment, the synthetic motion area is covered with the second local motion area to generate a synthetic frame, and then the method further includes: rendering the first frame image and the second frame image in a browser, so that the frame rate can be doubled.

[0034] This application also provides an application scenario to illustrate the technical solution of this application.

[0035] First, render the first frame of the image transmitted in the browser and save it as frame1.

[0036] Secondly, the difference between the transmitted second frame image frame2 and the first frame image frame1 is calculated, and the pixel with the largest difference is selected as the center, and the 200px×200px square region of interest is set as the second local motion region. Among them, 200px×200px is a default value and can be adjusted according to actual needs.

[0037] Then, the second local motion area motion_area2 of frame2 and the first local motion area motion_area1 corresponding to frame1 are sent to the trained shallow deep learning video interpolation model, and the loss function of the shallow deep learning video interpolation model is set to L1-loss.

[0038] The training datasets include conventional video interpolation datasets and self-made datasets. It should be noted that self-made datasets are generally various software operation videos. For example, if you need to interpolate an Adobe Photoshop video, you can use Adobe Photoshop operation videos to make a training dataset. If you need to interpolate a CAD video, you can use CAD operation videos to make a training dataset.

[0039] Finally, a 200×200 synthetic motion area is output, and the pixel values ​​of the synthetic motion area are used to cover the second local motion area motion_area2 corresponding to frame2 to obtain a new synthetic frame, thereby achieving the purpose of only updating the motion area, and then displaying the frame updated by the model, thereby displaying the latest transmitted frame.

[0040] Furthermore, by detecting whether the client has a GPU, the Wasm method or the webGPU method is selected to infer the shallow deep learning video interpolation model. If there is no GPU, the Wasm method is used, and if there is a GPU, the webGPU method is used.

[0041] The real-time video interpolation method based on deep learning model provided by this application was tested in Chrome browser. The whole test process was tested on a device with CPU I9-13900K and graphics card NVIDIA GeForce RTX 4090. The test video resolution was 1080P. In order to simplify the implementation and ensure the universality of the test results, in this experiment, wasm was used to interpolate the motion area. The experimental results are as follows: Figure 3 and Figure 4 As shown, from Figure 3 and Figure 4 From the above, the delay at 20fps is 68ms, and the delay at 40fps is 84ms, which means an increase of 16ms. The increased delay is mainly spent on the conversion between yuv420p and rgb, and model reasoning. It can be seen that these operations have little effect on the delay. This application only inserts frames in the moving area of ​​the video, and finally increases the frame rate from 20fps to 40fps, and keeps the video playback in the browser without significantly increasing the delay, which increases the smoothness of video viewing.

[0042] Based on the same inventive concept, the embodiment of the present application also provides a real-time video interpolation device based on a deep learning model for implementing the real-time video interpolation method based on a deep learning model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the real-time video interpolation device based on a deep learning model provided below can refer to the limitations of the real-time video interpolation method based on a deep learning model above, and will not be repeated here.

[0043] In an exemplary embodiment, a real-time video frame insertion device based on a deep learning model is provided, including the following modules.

[0044] The first local motion area and the second local motion area determination module are used to determine the first local motion area of ​​the first frame image and the second local motion area of ​​the second frame image according to the first frame image and the second frame image of the transmitted video; the second frame image is the next frame image of the first frame image.

[0045] The synthetic motion region output module is used to input the first local motion region and the second local motion region into a shallow deep learning video interpolation model and output a synthetic motion region.

[0046] The synthetic frame generation module is used to cover the second local motion area with the synthetic motion area to generate a synthetic frame.

[0047] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0048] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0049] The processor involved in each embodiment provided in this application may be a central processing unit or a graphics processor, but is not limited thereto.

[0050] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application; at the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A real-time video frame insertion method based on a deep learning model, characterized in that: The real-time video frame insertion method comprises: Determining a first local motion region of the first frame image and a second local motion region of the second frame image according to a first frame image and a second frame image of the transmitted video specifically includes: Decoding the first frame of image in a browser, and rendering the first frame of image; Decoding the second frame image in the browser, performing a difference between the second frame image and the first frame image, and selecting a pixel point with the largest difference as a center point to determine an area of ​​interest; the area of ​​interest is a second local motion area of ​​the second frame image; Corresponding the second local motion region to the first frame image to determine the first local motion region; the second frame image is a frame image next to the first frame image; Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthesized motion region, specifically comprising: the shallow deep learning video interpolation model comprises three convolutional layers; Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model to determine a first mask map and a second mask map, and performing warp operations on the first local motion region and the second local motion region and the optical flow field respectively to determine a first operation map and a second operation map; Performing a linear weighted summation process on the first operation map and the second operation map to determine a synthetic motion area; The synthetic motion region is covered with the second local motion region to generate a synthetic frame.

2. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Covering the second local motion area with the synthetic motion area to generate a synthetic frame specifically includes: Detect whether a GPU exists on the client of the browser, and obtain a first determination result; If the first judgment result is that the client of the browser has a GPU, the shallow deep learning video interpolation model is inferred by using a webGPU method, and the synthetic motion region is interpolated into the second local motion region to generate a synthetic frame; If the first judgment result is that there is no GPU on the client of the browser, the shallow deep learning video interpolation model is inferred using Wasm, and the synthetic motion area is interpolated into the second local motion area to generate a synthetic frame.

3. The real-time video frame insertion method based on deep learning model according to claim 1, characterized in that: Covering the second local motion area with the synthetic motion area to generate a synthetic frame, and then further comprising: The first frame image and the second frame image are rendered in a browser.

4. A real-time video frame insertion device based on a deep learning model, characterized in that: The real-time video frame insertion device based on the deep learning model includes: The first local motion region and the second local motion region determining module are used to determine the first local motion region of the first frame image and the second local motion region of the second frame image according to the first frame image and the second frame image of the transmitted video, and specifically include: Decoding the first frame of image in a browser, and rendering the first frame of image; Decoding the second frame image in the browser, performing a difference between the second frame image and the first frame image, and selecting a pixel point with the largest difference as a center point to determine an area of ​​interest; the area of ​​interest is a second local motion area of ​​the second frame image; Corresponding the second local motion region to the first frame image to determine the first local motion region; the second frame image is a frame image next to the first frame image; A synthetic motion region output module, used for inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model, and outputting a synthetic motion region, specifically comprising: the shallow deep learning video interpolation model includes three convolutional layers; Inputting the first local motion region and the second local motion region into a shallow deep learning video interpolation model to determine a first mask map and a second mask map, and performing warp operations on the first local motion region and the second local motion region and the optical flow field respectively to determine a first operation map and a second operation map; Performing a linear weighted summation process on the first operation map and the second operation map to determine a synthetic motion area; The synthetic frame generation module is used to cover the second local motion area with the synthetic motion area to generate a synthetic frame.

5. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the real-time video interpolation method based on a deep learning model described in any one of claims 1 to 3 is implemented.

Images