Video rendering method, video rendering device, and terminal device

Abstract

The present application provides a video rendering method, applied to the field of video processing. The video rendering method comprises the following steps: a video rendering device acquires a first video, the first video comprising an Nth image frame and an N+1th image frame; the video rendering device acquires a first instruction, the first instruction being used for operating a second video, and the second video being obtained by performing frame interpolation on the first video; and on the basis of the first instruction, the Nth image frame and the N+1th image frame, the video rendering device obtains a first interpolated frame image, wherein the second video comprises the Nth image frame, the N+1th image frame, and the first interpolated frame image, and the first interpolated frame image is an image after the N+1th image frame in the second video. In the present solution provided by the present application, an interpolated frame image is generated by means of extrapolation, so that the delay between the video rendering device and a terminal device can be reduced. In addition, the interpolated frame image is generated by means of the instruction for the second video, so that the quality of the interpolated frame image can be improved, thereby improving user experience.

Description

Video rendering method, video rendering device and terminal device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 22, 2024, with application number 202410350166.X and application name “Video rendering method, video rendering device and terminal device”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on June 13, 2024, with application number 202410767545.9 and application name “Video rendering method, video rendering device and terminal device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of video processing, and in particular to a video rendering method, a video rendering device, and a terminal device. Background Art

[0003] Cloud rendering refers to the use of cloud device resources to render 3D graphics or animations on a terminal device. Cloud rendering provides a high-computing, low-cost service for terminal devices. Cloud rendering includes frame interpolation, which inserts new images into the original video. Specifically, the original video consists of two consecutive frames. Frame interpolation involves inserting a new image between these two frames. However, in some scenarios, such as cloud gaming, cloud devices need to perform real-time rendering based on terminal device instructions. In these scenarios, the rendering process of the cloud device directly affects the latency between the cloud device and the terminal device. High network latency can cause game lag or tearing, reducing the user experience. Summary of the Invention

[0004] This application provides a video rendering method, a video rendering device, and a terminal device. By generating interpolated images through extrapolation, the delay between the video rendering device and the terminal device can be reduced. Furthermore, by generating interpolated images based on instructions for a second video, the quality of the interpolated images can be improved, thereby enhancing the user experience.

[0005] The first aspect of the present application provides a video rendering method. The video rendering method is applied to a video rendering device. The video rendering device can be a cloud device, such as a server. The video rendering method includes the following steps: the video rendering device obtains a first video, the first video includes the Nth frame image and the N+1th frame image, N is an integer greater than 0; the video rendering device obtains a first instruction, the first instruction instructs to operate on the second video, the second video is obtained after interpolation of the first video; the video rendering device obtains a first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, wherein the second video includes the Nth frame image, the N+1th frame image and the first interpolated frame image, and the first interpolated frame image is an image after the N+1th frame image in the second video.

[0006] In an optional embodiment of the first aspect, the video rendering device obtaining the first interpolated image based on the first instruction, the Nth frame image, and the N+1th frame image includes: the video rendering device performing optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow. By using the instruction for the second video in the optical flow estimation, the accuracy of the optical flow estimation can be improved, image quality can be improved, and thus the user experience can be enhanced.

[0007] In an optional manner of the first aspect, the first video further includes an N-1th frame image. The second video further includes a second interpolated frame image. The second interpolated frame image is obtained based on the N-1th frame image and the Nth frame image. The second interpolated frame image is located between the Nth frame image and the N+1th frame image in the second video. The first instruction includes a target instruction. The target instruction instructs to operate on the second interpolated frame image and the N+1th frame image. The video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, which includes: the video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image according to the target instruction to obtain the first optical flow. By using the target instruction to perform optical flow estimation, the accuracy of the optical flow estimation can be improved, thereby improving the quality of the obtained interpolated frame image.

[0008] In an optional embodiment of the first aspect, the video rendering device does not use instructions other than the target instruction to generate the first interpolated frame image. By using the target instruction for optical flow estimation, the efficiency of optical flow estimation can be improved, thereby reducing the latency between the video rendering device and the terminal device and improving the user experience.

[0009] In an optional manner of the first aspect, the video rendering device obtaining the first interpolated image based on the first instruction, the Nth frame image, and the N+1th frame image includes: the video rendering device performing an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated image. Obtaining the interpolated image through optical flow estimation and affine transformation can improve the quality of the obtained interpolated image.

[0010] In an optional embodiment of the first aspect, the video rendering method further includes the following steps: the video rendering device obtains a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image. The video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, which includes: the video rendering device performs optical flow estimation on the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image according to the first instruction to obtain the first optical flow. The video rendering method further includes the following steps: the video rendering device upsamples the first optical flow to obtain a second optical flow; the video rendering device super-resolutions the downsampled image of the N+1th frame image to obtain a first image of the N+1th frame image; and the video rendering device performs an affine transformation on the N+1th frame image and the first optical flow to obtain a first interpolated image, which includes: the video rendering device performs an affine transformation on the first image and the second optical flow to obtain the first interpolated image. The parallel processing of super-resolution and optical flow estimation can improve the efficiency of the video rendering device in obtaining interpolated images, thereby reducing the latency between the video rendering device and the terminal device and improving the user experience.

[0011] In an optional manner of the first aspect, the video rendering method further includes the following steps: the video rendering device obtains a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image; the video rendering device super-resolutions the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image to obtain a first image of the Nth frame image and a second image of the N+1th frame image. The video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, which includes: the video rendering device performs optical flow estimation on the first image and the second image according to the first instruction to obtain the first optical flow. By interpolating the downsampled images, the hardware requirements for the video rendering device can be reduced, thereby reducing the hardware cost of the video rendering device.

[0012] In an optional manner of the first aspect, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. In actual applications, the first interpolated frame image can be the second frame or third frame or other later image after the N+1th frame image. However, since there are other frame images between the first interpolated frame image and the N+1th frame image, the quality of the first interpolated frame image obtained by the video rendering device will be low. Therefore, the present application can improve the quality of the obtained interpolated frame image. In an optional manner of the first aspect, the N+1th frame image is the last frame image in the first video.

[0013] In an optional manner of the first aspect, N is equal to 1. In the second video, no other images are included between the Nth frame image and the N+1th frame image.

[0014] The second aspect of the present application provides a video rendering device. The video rendering device includes a first acquisition unit, a second acquisition unit and an interpolation unit. The first acquisition unit is used to acquire a first video. The first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second acquisition unit is used to acquire a first instruction. The first instruction instructs to operate on the second video, and the second video is obtained after interpolation of the first video. The interpolation unit is used to obtain a first interpolation image based on the first instruction, the Nth frame image and the N+1th frame image. The second video includes the Nth frame image, the N+1th frame image and the first interpolation image. The first interpolation image is an image after the N+1th frame image in the second video.

[0015] In an optional manner of the second aspect, the interpolation unit is used to obtain a first interpolation image based on the first instruction, the Nth frame image and the N+1th frame image, including: the interpolation unit is used to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain a first optical flow.

[0016] In an optional manner of the second aspect, the first video further includes an N-1th frame image, the second video further includes a second interpolated frame image, the second interpolated frame image is obtained based on the N-1th frame image and the Nth frame image, and the second interpolated frame image is located between the Nth frame image and the N+1th frame image in the second video, and the first instruction includes a target instruction, the target instruction instructing to operate on the second interpolated frame image and the N+1th frame image. The interpolation unit is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction, and obtaining the first optical flow includes: the interpolation unit is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the target instruction to obtain the first optical flow.

[0017] In an optional manner of the second aspect, the interpolation unit is used to obtain a first interpolation image based on the first instruction, the Nth frame image and the N+1th frame image, including: the interpolation unit is used to perform an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolation image.

[0018] In an optional manner of the second aspect, the interpolation unit is further used to obtain a downsampled image of the N-th frame image and a downsampled image of the N+1-th frame image. The interpolation unit is used to perform optical flow estimation on the N-th frame image and the N+1-th frame image according to the first instruction, and obtaining the first optical flow includes: the interpolation unit is used to perform optical flow estimation on the downsampled image of the N-th frame image and the downsampled image of the N+1-th frame image according to the first instruction, and obtain the first optical flow. The interpolation unit is also used to upsample the first optical flow to obtain the second optical flow. The interpolation unit is also used to super-resolution the downsampled image of the N+1-th frame image to obtain the first image of the N+1-th frame image. The interpolation unit is used to perform affine transformation on the N+1-th frame image and the first optical flow, and obtaining the first interpolated image includes: the interpolation unit is used to perform affine transformation on the first image and the second optical flow to obtain the first interpolated image.

[0019] In an optional manner of the second aspect, the interpolation unit is further used to obtain a downsampled image of the N-th frame image and a downsampled image of the N+1-th frame image. The interpolation unit is further used to super-resolve the downsampled image of the N-th frame image and the downsampled image of the N+1-th frame image to obtain a first image of the N-th frame image and a second image of the N+1-th frame image. The interpolation unit is used to perform optical flow estimation on the N-th frame image and the N+1-th frame image according to the first instruction, and obtaining the first optical flow includes: the interpolation unit is used to perform optical flow estimation on the first image and the second image according to the first instruction, and obtain the first optical flow. The interpolation unit is used to perform an affine transformation on the N+1-th frame image and the first optical flow, and obtaining the first interpolated image includes: the interpolation unit is used to perform an affine transformation on the second image and the first optical flow, and obtain the first interpolated image.

[0020] In an optional manner of the second aspect, the first interpolated frame image is the next frame image of the N+1th frame image in the second video.

[0021] The third aspect of the present application provides a video rendering device. The video rendering device includes a transceiver and a processor. The processor is used to obtain a first video, and the first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The processor is also used to obtain a first instruction, and the first instruction instructs to operate on the second video, and the second video is obtained after inserting a frame into the first video. The processor is also used to obtain a first inserted frame image based on the first instruction, the Nth frame image, and the N+1th frame image. The second video includes the Nth frame image, the N+1th frame image, and the first inserted frame image, and the first inserted frame image is an image after the N+1th frame image in the second video. The transceiver is used to send the second video to the terminal device.

[0022] In an optional manner of the third aspect, the processor is used to obtain a first interpolated image based on the first instruction, the Nth frame image and the N+1th frame image, including: the processor is used to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain a first optical flow.

[0023] In an optional manner of the third aspect, the first video further includes an N-1th frame image, the second video further includes a second interpolated frame image, the second interpolated frame image is obtained based on the N-1th frame image and the Nth frame image, and the second interpolated frame image is located between the Nth frame image and the N+1th frame image in the second video, and the first instruction includes a target instruction, the target instruction instructing an operation to be performed on the second interpolated frame image and the N+1th frame image. The processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction, and obtaining the first optical flow includes: the processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the target instruction to obtain the first optical flow.

[0024] In an optional manner of the third aspect, the processor is used to obtain a first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, including: the processor is used to perform an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated frame image.

[0025] In an optional manner of the third aspect, the processor is further configured to obtain a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image. The processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction, and obtaining the first optical flow includes: the processor is configured to perform optical flow estimation on the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image according to the first instruction, and obtain the first optical flow. The processor is also configured to upsample the first optical flow to obtain the second optical flow. The processor is also configured to super-resolution the downsampled image of the N+1th frame image to obtain the first image of the N+1th frame image. The processor is configured to perform an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated frame image, including: the processor is configured to perform an affine transformation on the first image and the second optical flow to obtain the first interpolated frame image.

[0026] In an optional manner of the third aspect, the processor is further configured to obtain a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image. The processor is further configured to super-resolve the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image to obtain a first image of the Nth frame image and a second image of the N+1th frame image. The processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction, and obtaining the first optical flow includes: the processor is configured to perform optical flow estimation on the first image and the second image according to the first instruction, and obtain the first optical flow. The processor is configured to perform an affine transformation on the N+1th frame image and the first optical flow, and obtaining the first interpolated frame image includes: the processor is configured to perform an affine transformation on the second image and the first optical flow, and obtain the first interpolated frame image.

[0027] In an optional manner of the third aspect, the first interpolated frame image is the next frame image of the N+1th frame image in the second video.

[0028] The fourth aspect of the present application provides a video acquisition method. The video acquisition method is applied to a terminal device. The terminal device can be a PC, a mobile phone, a tablet, etc. The video acquisition method includes the following steps: the terminal device receives a second video from a video rendering device, the second video is obtained by the video rendering device inserting frames into the first video, the first video includes the Nth frame image and the N+1th frame image, N is an integer greater than 0, the second video includes the Nth frame image, the N+1th frame image and the first inserting frame image, the first inserting frame image is the image after the N+1th frame image in the second video, the first inserting frame image is obtained based on the first instruction, the Nth frame image and the N+1th frame image, the first instruction indicates to operate the second video; the terminal device displays the second video.

[0029] In an optional manner of the fourth aspect, the video acquisition method includes the following steps: the terminal device sends a first instruction to the video rendering device.

[0030] The fifth aspect of the present application provides a terminal device. The terminal device includes a receiving unit and a display unit. The receiving unit is used to receive a second video from a video rendering device. The second video is obtained by the video rendering device interpolating the first video, and the first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second video includes the Nth frame image, the N+1th frame image, and the first interpolated frame image. The first interpolated frame image is an image after the N+1th frame image in the second video, and the first interpolated frame image is obtained based on a first instruction, the Nth frame image, and the N+1th frame image, and the first instruction indicates an operation on the second video. The display unit is used to display the second video.

[0031] The sixth aspect of the present application provides a terminal device. The terminal device includes a transceiver and a display. The transceiver is used to receive a second video from a video rendering device. The second video is obtained by the video rendering device interpolating the first video. The first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second video includes the Nth frame image, the N+1th frame image, and the first interpolated frame image. The first interpolated frame image is an image after the N+1th frame image in the second video, and the first interpolated frame image is obtained based on a first instruction, the Nth frame image, and the N+1th frame image, and the first instruction indicates an operation on the second video. The display is used to display the second video.

[0032] The seventh aspect of the present application provides a video rendering system. The video rendering system includes a video rendering device and a terminal device. The video rendering device is used to obtain a first video. The first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The video rendering device is also used to obtain a first instruction, the first instruction instructing to operate on the second video, and the second video is obtained after interpolation of the first video. The video rendering device is also used to obtain a first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image. The second video includes the Nth frame image, the N+1th frame image and the first interpolated frame image. The first interpolated frame image is an image after the N+1th frame image in the second video. The video rendering device is also used to send the second video to the terminal device. The terminal device is used to display the second video.

[0033] It should be understood that the description of the fourth aspect, the fifth aspect, the sixth aspect or the seventh aspect is similar to the description of the first aspect or any one of the optional methods in the first aspect. Therefore, the description of the fourth aspect, the fifth aspect, the sixth aspect or the seventh aspect may refer to the description of the video rendering method in the first aspect or any one of the optional methods in the first aspect. For example, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. For another example, the terminal device is further used to send a first instruction for the second video to the video rendering device. The video rendering device is used to obtain the first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image.

[0034] In an eighth aspect, the present application provides a chip including a processing circuit configured to execute the method described in the first aspect, any optional manner of the first aspect, the fourth aspect, or any optional manner of the fourth aspect.

[0035] In a ninth aspect, the present application provides a computer-readable storage medium storing instructions. When the instructions are executed by a computer, the method described in the first aspect, any optional method of the first aspect, or the fourth aspect is implemented.

[0036] The tenth aspect of the present application provides a computer program product, which includes instructions. When the instructions are run on a computer, the computer executes the method described in the first aspect, any optional method of the first aspect, or the fourth aspect.

[0037] Among them, the technical effects brought about by the second to tenth aspects or any optional methods therein can refer to the technical effects brought about by the above-mentioned first aspect and different optional methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a first flow chart of a video rendering method provided by an embodiment of the present application;

[0039] FIG2 is a schematic diagram of the structure of a first video provided in an embodiment of the present application;

[0040] FIG3 is a first structural diagram of a first video and a second video provided in an embodiment of the present application;

[0041] FIG4 is a second structural diagram of the first video and the second video provided in an embodiment of the present application;

[0042] FIG5 is a third structural diagram of the first video and the second video provided in an embodiment of the present application;

[0043] FIG6 is a second flow chart of the video rendering method provided in an embodiment of the present application;

[0044] FIG7a is a fourth structural diagram of the first video and the second video provided in an embodiment of the present application;

[0045] FIG7 b is a fifth structural diagram of the first video and the second video provided in an embodiment of the present application;

[0046] FIG8a is a first correspondence diagram between an operation instruction and a second video provided in an embodiment of the present application;

[0047] FIG8b is a second corresponding relationship diagram between the operation instruction and the second video provided in an embodiment of the present application;

[0048] FIG9 is a diagram showing the correspondence between the first video, the second video, and the operation instruction flow provided in an embodiment of the present application;

[0049] FIG10 is a flow chart of a video acquisition method according to an embodiment of the present application;

[0050] FIG11 is a first structural diagram of a video rendering device provided in an embodiment of the present application;

[0051] FIG12 is a second structural diagram of the video rendering device provided in an embodiment of the present application;

[0052] FIG13 is a third structural diagram of the video rendering device provided in an embodiment of the present application;

[0053] FIG14 is a fourth structural diagram of a video rendering device provided in an embodiment of the present application;

[0054] FIG15 is a first structural diagram of a terminal device provided in an embodiment of the present application;

[0055] FIG16 is a second structural diagram of a terminal device provided in an embodiment of the present application;

[0056] FIG17 is a third structural diagram of a terminal device provided in an embodiment of the present application;

[0057] FIG18 is a fourth structural diagram of a terminal device provided in an embodiment of the present application;

[0058] FIG19 is a schematic structural diagram of a video rendering system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application provides a video rendering method, a video rendering device, and a terminal device, which generate interpolated images by extrapolation, and can reduce the delay between the video rendering device and the terminal device, thereby improving the user experience. Moreover, by generating interpolated images based on instructions for the second video, the quality of the interpolated images can be improved, thereby improving the user experience. It should be understood that the "first", "second", "target", etc. used in this application are only used for the purpose of distinguishing the description, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, for the sake of simplicity and clarity, reference numbers and / or letters are repeated in multiple figures of this application. Repetition does not indicate a strict limiting relationship between various embodiments and / or configurations.

[0060] Cloud rendering refers to the terminal device using the resources of the cloud device to render 3D graphics or animations. Cloud rendering includes interpolation. Interpolation refers to inserting a new image into the original video. The prior art provides an interpolation technology based on deep learning super sampling (DLSS). Specifically, the cloud device obtains two consecutive frames of images in the original video, generates an interpolated image based on the two consecutive frames of images through DLSS technology, and sets the interpolated image between the two consecutive frames of images. All two consecutive frames of images in the original video are repeatedly processed in the above manner to obtain a new video. Among them, the two consecutive frames of images include the previous frame image and the next frame image. Before generating the interpolated image by interpolation, the cloud device must first obtain the next frame image, thereby reducing the efficiency of the cloud device in obtaining the interpolated image and increasing the delay between the terminal device and the cloud device.

[0061] To this end, the present application provides a video rendering method. The video rendering method is applied to a video rendering device. The video rendering device can be a cloud device in end-cloud collaboration, such as a server. The video rendering device can also be a terminal device, such as a PC, mobile phone or tablet. Figure 1 is a first flow chart of the video rendering method provided in an embodiment of the present application. As shown in Figure 1, the video rendering method includes the following steps.

[0062] In step 101, a video rendering device obtains a first video, where the first video includes an Nth frame image and an N+1th frame image.

[0063] The video rendering device can obtain the first video in any of the following ways. In one way, the video rendering device receives the first video from the terminal device. In another way, the video rendering device generates the first video based on the request of the terminal device in the end-cloud collaboration. For example, in the cloud gaming scenario, when the terminal device starts running the cloud game, it sends a game screen display request to the video rendering device. The screen display request can carry rendering parameters, such as the resolution, refresh rate range, rendering resolution, or rendering ratio of the terminal device. In a multiplayer game mode, the video rendering device may simultaneously receive multiple game screen display requests from multiple terminal devices. Multiple terminal devices and multiple game screen display requests correspond one-to-one. The video rendering device generates the first video based on the game screen display request. The first video includes multiple frames of images. The multiple frames of images include consecutive N-th frame images and N+1-th frame images. N is an integer greater than 0. Figure 2 is a schematic diagram of the structure of the first video provided in an embodiment of the present application. As shown in Figure 2, the box in the figure represents a frame of image. The first video includes multiple frames of images, and the multiple frames of images are sorted in a specified order, for example, in the order in which the images are generated or in the order in which the videos are played.

[0064] In step 102, the video rendering device obtains a first instruction, where the first instruction instructs to operate on a second video, where the second video is obtained by inserting frames into the first video.

[0065] The second video is obtained by the video rendering device after inserting frames into the first video. For the description of the second video, please refer to the description of the subsequent step 103. After obtaining the second video by inserting frames, the video rendering device sends the second video to the terminal device. The first instruction indicates the operation of the second video. For example, the terminal device displays the second video through a display screen, and receives the user's operation instructions for the characters in the second video through the display screen, such as moving forward, turning left, or jumping. The terminal device sends the operation instruction to the video rendering device. The video rendering device uses the operation instruction as the first instruction. Alternatively, the video rendering device converts the operation instruction and obtains the first instruction. Therefore, the operation instruction can also be called the first instruction. For another example, the terminal device displays the second video through a display screen and receives the operation instruction through a joystick.

[0066] In step 103, the video rendering device obtains a first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, wherein the second video includes the Nth frame image, the N+1th frame image and the first interpolated frame image, and the first interpolated frame image is the image after the N+1th frame image in the second video.

[0067] Before interpolating frames, the video rendering device needs to generate an interpolated image based on the original image in the first video. After generating the interpolated image, the video rendering device inserts the interpolated image into the original image, thereby obtaining a second video. After obtaining the second video, the video rendering device can encode the second video and send the encoded second video to the terminal device. In the second video, the interpolated image is located after the original image from which the interpolated image was generated. For example, the original image includes the Nth frame image and the N+1th frame image. The video rendering device generates a first interpolated image based on the first instruction, the Nth frame image, and the N+1th frame image. The first interpolated image is the image after the N+1th frame image in the second video. Figure 3 is a first structural schematic diagram of the first and second videos provided in an embodiment of the present application. As shown in Figure 3, the first video includes a consecutive Nth frame image and the N+1th frame image. The Nth frame image corresponds to N+0 in Figure 3. The N+1th frame image corresponds to N+1 in Figure 3. The second video includes the Nth frame image, the N+1th frame image, and the first interpolated image. The first interpolated image corresponds to n+1 in Figure 3. The first interpolated image follows the Nth and N+1th frames. In subsequent examples of this application, n represents an interpolated image, and N represents the original image. The first video includes the original image. The second video includes the original image and the interpolated image. It should be understood that Figure 3 only shows a portion of the images in the first and second videos. In actual applications, the first and second videos may include more images. Similarly, in subsequent illustrations, only a portion of the images in the video may be shown.

[0068] In an embodiment of the present application, an interpolated image is generated by extrapolation. At this time, the video rendering device does not need to wait for the N+1 frame image to be generated, or wait for the terminal device to send the N+1 frame image. Therefore, the embodiment of the present application improves the efficiency of the video rendering device in obtaining the interpolated image and reduces the delay between the terminal device and the video rendering device. In addition, by generating the interpolated image based on the instruction of the second video, the quality of the interpolated image can be improved, thereby improving the user experience.

[0069] Figure 4 is a second structural schematic diagram of the first video and the second video provided in an embodiment of the present application. As shown in Figure 4, the first video includes the N+0th to N+5th frame images. The second video includes the N+0th to N+5th frame images and the n+1th to n+5th frame images. The N+0th to N+5th frame images are the original images of the second video. The n+1th to n+5th frame images are interpolated images of the second video. The ordering method of the original images and the interpolated images is shown in Figure 4. Among them, the n+1th frame image is obtained by the video rendering device based on the Nth frame image and the N+1th frame image. The n+1th frame image is located between the N+2th frame image and the N+3th frame image in the second video. Similarly, the n+2th frame image is obtained by the video rendering device based on the N+1th frame image and the N+2th frame image. The n+2th frame image is located between the N+3th frame image and the N+4th frame image in the second video. The generation relationship between other interpolated images and the original image can be referred to the solid arrows in the figure. The order relationship between other interpolated images and the original image can be referred to the order of images in the figure.

[0070] In the example of Figure 4 , the n+1th frame image in the second video is the frame image next to the N+1th frame image. In practical applications, the n+1th frame image in the second video may be the X frames image next to the N+1th frame image. X is an integer greater than or equal to 1. When the value of X is larger, the quality of the first interpolated frame image obtained by the video rendering device may be lower. Therefore, embodiments of the present application can improve the quality of the obtained interpolated frame images.

[0071] In the example of Figure 4, the n+1th frame image is obtained based on two frames of original images. In actual applications, the video rendering device can obtain interpolated images based on more frames of original images. For example, the video rendering device can obtain interpolated images based on three frames of original images. Figure 5 is a third structural schematic diagram of the first video and the second video provided in an embodiment of the present application. As shown in Figure 5, the first video includes the N+0th frame image to the N+5th frame image. The second video includes the N+0th frame image to the N+5th frame image and the n+2th frame image to the n+5th frame image. The N+0th frame image to the N+5th frame image are the original images of the second video. The n+2th frame image to the n+5th frame image are the interpolated images of the second video. The ordering method of the original images and the interpolated images is shown in Figure 5. Among them, the n+2th frame image is obtained by the video rendering device based on the Nth frame image, the N+1th frame image and the N+2th frame image. Similarly, the n+3th frame image is obtained by the video rendering device based on the N+1th frame image, the N+2th frame image, and the N+3th frame image. The relationship between other interpolated frame images and the original image can be seen from the solid arrows in the figure.

[0072] It should be understood that n in the diagram can be a negative value. For example, when N is 1, n+2 in Figure 4 represents the first frame in the interpolated image. In this case, n is equal to -1. It should be understood that the values ​​within the boxes in the diagram are merely one way to distinguish between different images. In actual applications, those skilled in the art can adapt the values ​​within the boxes as needed.

[0073] In the example of FIG. 4 , no interpolated frame image is included between the Nth frame image and the N+1th frame image of the second video. Interpolated frame images are included between other two adjacent original images of the second video. In practical applications, to maintain a stable frame rate, the video rendering device may insert a target frame image between the Nth frame image and the N+1th frame image. The target frame image may be the Nth frame image or the N+1th frame image. Alternatively, the target frame image may be obtained based on the Nth frame image or the N+1th frame image.

[0074] As can be seen from the foregoing description, the video rendering device in this application generates interpolated images by extrapolation. The following describes an example of how the video rendering device generates interpolated images, taking the example of a video rendering device obtaining a first interpolated image based on the Nth frame image and the N+1th frame image. For example, the video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image based on a first instruction to obtain a first optical flow. Optical flow refers to the temporal motion of each pixel in an image. In an embodiment of the present application, optical flow estimation is used to predict the motion information of an object in an image to determine the position information of the object at a certain moment in the future. Optical flow estimation can be dense optical flow estimation or sparse optical flow estimation. Dense optical flow estimation refers to calculating the optical flow vector at every pixel in the image, while sparse optical flow estimation only selects some pixels to calculate the optical flow vector. The algorithm used for optical flow estimation can be optical flow estimation based on brightness changes, optical flow estimation based on image matching, or optical flow estimation based on a convolutional neural network. The video rendering device performs an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated image. Affine transformation is also known as image warping. The first optical flow represents the position of an object at a certain moment in the future. By applying an affine transformation to the N+1th frame using the first optical flow, we can predict how the N+1th frame will change in the future, which is the first interpolated frame.

[0075] In practical applications, to reduce the performance requirements of the video rendering device, the video rendering device can obtain an interpolated image based on a downsampled image of the original image. This is described below. Figure 6 is a second flow chart of the video rendering method provided in an embodiment of the present application. As shown in Figure 6, the video rendering method includes the following steps.

[0076] In step 601 , a video rendering device obtains a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image.

[0077] The video rendering device obtains a downsampled image of the Nth frame by downsampling the Nth frame image. Downsampling is also called reducing the image or downsampling. The embodiment of the present application does not limit the downsampling algorithm used by the video rendering device. Similarly, the video rendering device obtains a downsampled image of the N+1th frame image by downsampling the N+1th frame image.

[0078] In step 602, the video rendering device performs super-resolution on the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image to obtain a first image of the Nth frame image and a second image of the N+1th frame image.

[0079] The video rendering device obtains the first image of the Nth frame image by super-resolution on the downsampled image of the Nth frame image. Super-resolution (SR) is to restore a high-resolution image from a low-resolution image or video sequence. The super-resolution algorithm can be an interpolation, reconstruction or an algorithm based on a god-level network. The embodiment of the present application does not limit the algorithm used by the video rendering device to perform super-resolution. Similarly, the video rendering device obtains the second image of the N+1th frame image by super-resolution on the downsampled image of the N+1th frame image.

[0080] In step 603, the video rendering device performs optical flow estimation on the first image and the second image to obtain a first optical flow.

[0081] In step 604 , the video rendering device performs affine transformation on the second image and the first optical flow to obtain a first interpolated image.

[0082] Figure 7a is a fourth structural diagram of the first video and the second video provided in an embodiment of the present application. As shown in Figure 7a, the first video includes the N+0th frame image to the N+5th frame image. The video rendering method is described below based on Figure 7a.

[0083] The video rendering device downsamples the first video to obtain a sampling thumbnail. As shown in Figure 7a, the sampling thumbnail includes the N1+0th frame image to the N1+5th frame image. Among them, the N1+0th frame image is obtained by the video rendering device downsampling the N+0th frame image. Similarly, the N1+0th frame image is obtained by the video rendering device downsampling the N+1th frame image. The generation relationship between each image in the sampling thumbnail and the image in the first video can be referred to the solid arrows in the figure. The order relationship between each image in the sampling thumbnail can be referred to the order of the images in the figure.

[0084] The video rendering device super-resolutions the sampled image to obtain a high-resolution image, referred to as a high-resolution image. As shown in Figure 7a, the high-resolution image includes the N2+0th frame image to the N2+5th frame image. Among them, the N2+0th frame image is obtained by the video rendering device super-resolutioning the N1+0th frame image. Similarly, the N2+0th frame image is obtained by the video rendering device super-resolutioning the N1+1th frame image. The generation relationship between each image in the high-resolution image and the image in the sampled image can be referred to the solid arrows in the figure. The ordering relationship between each image in the high-resolution image can be referred to the ordering of the images in the figure.

[0085] The video rendering device performs optical flow estimation on adjacent images in the high-resolution image to obtain an optical flow image. As shown in Figure 7a, the optical flow image includes the N3+1th frame image to the N3+5th frame image. Among them, the N3+1th frame image is obtained by the video rendering device performing optical flow estimation on the N2+0th frame image and the N2+1th frame image. Similarly, the N3+2th frame image is obtained by the video rendering device performing optical flow estimation on the N2+1th frame image and the N2+2th frame image. The generation relationship between each image in the optical flow image and the image in the high-resolution image can be referred to the solid arrows in the figure. The sorting relationship between each image in the optical flow image can be referred to the sorting of images in the figure.

[0086] The video rendering device performs an affine transformation on the optical flow image and the high-resolution image to obtain an interpolated image. As shown in Figure 7a, the interpolated image includes the n+1th frame image to the n+5th frame image. Among them, the n+1th frame image is obtained by the video rendering device performing an affine transformation on the N2+1th frame image based on the N3+1th frame image. Similarly, the n+2th frame image is obtained by the video rendering device performing an affine transformation on the N2+2th frame image based on the N3+2th frame image. The generation relationship between each image in the interpolated image and the images in the high-resolution image and the optical flow image can be referred to the solid arrows in the figure. The sorting relationship between each image in the interpolated image can be referred to the sorting of images in the figure.

[0087] The video rendering device synthesizes the interpolated image and the first video to produce a second video. As shown in Figure 7a, the second video includes images from frames N+0 to N+5 and frames n+1 to n+5. Frame n+1 is located between frames N+1 and N+2. Similarly, frame n+2 is located between frames N+2 and N+3. The ordering of the images in the second video can be referenced in the image ordering diagram.

[0088] In the examples of Figures 6 and 7a, the video rendering device first performs super-resolution processing and then performs optical flow estimation. Therefore, in the scenario of end-cloud collaboration, the above method will increase the time it takes for the video rendering device to obtain interpolated images, thereby increasing the delay between the video rendering device and the terminal device and reducing the user experience. In the video rendering method provided in this application, the video rendering device can process super-resolution and optical flow estimation in parallel, improve the efficiency of the video rendering device in obtaining interpolated images, and reduce the delay between the video rendering device and the terminal device. This is described below with an example.

[0089] FIG7b is a fifth structural diagram of the first video and the second video provided in an embodiment of the present application. As shown in FIG7b, the first video includes the N+0th frame image to the N+5th frame image. The video rendering method is described below based on FIG7b.

[0090] The video rendering device downsamples the first video to obtain a sampling thumbnail. As shown in Figure 7b, the sampling thumbnail includes the N1+0th frame image to the N1+5th frame image. Among them, the N1+0th frame image is obtained by the video rendering device downsampling the N+0th frame image. Similarly, the N1+0th frame image is obtained by the video rendering device downsampling the N+1th frame image. The generation relationship between each image in the sampling thumbnail and the image in the first video can be referred to the solid arrows in the figure. The sorting relationship between each image in the sampling thumbnail can be referred to the sorting of images in the figure.

[0091] The video rendering device performs optical flow estimation on adjacent images in the sampling sub-image to obtain optical flow image 1. As shown in Figure 7b, optical flow image 1 includes the N3+1th frame image to the N3+5th frame image. Among them, the N3+1th frame image is obtained by the video rendering device performing optical flow estimation on the N1+0th frame image and the N1+1th frame image. Similarly, the N3+2th frame image is obtained by the video rendering device performing optical flow estimation on the N1+1th frame image and the N1+2th frame image. The generation relationship between each image in the optical flow image 1 and the image in the sampling sub-image can be referred to the solid arrows in the figure. The sorting relationship between each image in the optical flow image 1 can be referred to the sorting of images in the figure.

[0092] The video rendering device upsamples the optical flow image 1 to obtain the optical flow image 2. As shown in Figure 7b, the optical flow image 2 includes the N4+1th frame image to the N4+5th frame image. The N4+1th frame image is obtained by the video rendering device upsampling the N3+1th frame image. Similarly, the N4+2th frame image is obtained by the video rendering device upsampling the N3+2th frame image. The generation relationship between the images in the optical flow image 2 and the optical flow image 1 can be seen from the solid arrows in the figure. The ordering relationship between the images in the optical flow image 2 can be seen from the ordering of the images in the figure.

[0093] The video rendering device super-resolutions the sampled small image to obtain a high-resolution image, referred to as a high-resolution image. As shown in Figure 7b, the high-resolution image includes the N2+0th frame image to the N2+5th frame image. Among them, the N2+0th frame image is obtained by the video rendering device super-resolutioning the N1+0th frame image. Similarly, the N2+0th frame image is obtained by the video rendering device super-resolutioning the N1+1th frame image. The generation relationship between each image in the high-resolution image and the image in the sampled small image can be referred to the solid arrows in the figure. The sorting relationship between each image in the high-resolution image can be referred to the sorting of images in the figure.

[0094] The video rendering device performs an affine transformation on the optical flow image 2 and the high-resolution image to obtain an interpolated image. The video rendering device synthesizes the interpolated image with the first video to obtain a second video. It should be understood that the description of the interpolated image, the first video, and the second video in Figure 7b is similar to the description of the interpolated image, the first video, and the second video in Figure 7a . Therefore, the description of the interpolated image, the first video, and the second video in Figure 7b can refer to the description of the interpolated image, the first video, and the second video in Figure 7a . After generating the second video, the video rendering device can send the second video to a terminal device. The terminal device is used to display the second video. During the display of the second video, the terminal device can generate operation instructions, such as a user operating the image displayed in the second video through the terminal device's touch screen. The operation instructions are described below using the second video in Figure 4 as an example. Figure 8a is a first correspondence diagram between operation instructions and the second video provided in an embodiment of the present application. As shown in Figure 8a , during the display of the second video by the terminal device, the terminal device can generate an operation instruction stream. In the time domain, the second video can be divided into multiple images. For example, in the example of Figure 8a, the second video can be divided into 10 images. Similarly, the operation instruction stream can also be divided in the time domain. In the example of Figure 8a, the operation instruction stream includes 10 operation instructions. The 10 operation instructions are operation instructions 1 to operation instructions 10. The 10 operation instructions correspond one-to-one with the 10 images. Operation instruction 1 corresponds to the N+0th frame image. Similarly, operation instruction 2 corresponds to the N+1th frame image. The correspondence between operation instructions and images in the second video can be referenced by the position of the dotted line in the figure. Operation instruction 1 represents the operation instruction generated by the terminal device when displaying the N+0th frame image. For example, when the terminal device is displaying the N+0th frame image, the user issues a forward instruction to the character in the image via the touch screen. It should be understood that operation instruction 1 may include one or more instructions. For example, when the terminal device is displaying the N+0th frame image, the user issues a forward instruction and a jump instruction to the character in the image via the touch screen.

[0095] It should be understood that Figure 8a is a correspondence diagram between the operation instructions provided in an embodiment of the present application and the second video. In practical applications, those skilled in the art can establish a correspondence diagram between the operation instructions and the second video as needed. For example, Figure 8b is a second correspondence diagram between the operation instructions provided in an embodiment of the present application and the second video. As shown in Figure 8b, the second video can be divided into 10 images. The operation instruction stream includes 5 operation instructions in the time domain. The 5 operation instructions are operation instructions 1 to operation instructions 5. The 5 operation instructions correspond one-to-one with two images in the 10 images. Operation instruction 1 corresponds to the N+0th frame image and the N+1th frame image. Similarly, operation instruction 2 corresponds to the n+1th frame image and the N+2th frame image. The correspondence between the operation instructions and the images in the second video can be referenced by the position of the dotted line in the figure. Operation instruction 1 represents the operation instruction generated by the terminal device when displaying the N+0th frame image and the N+1th frame image. For example, when the terminal device is displaying the N+0th frame image and the N+1th frame image, the user issues a forward instruction to the character in the image via the touch screen.

[0096] In actual applications, the video rendering device generates the first video and the second video, the terminal device displays the second video, and generates the operation instructions, which can all be a real-time process that continues over a period of time. At this time, a closed loop is formed between the video rendering device and the terminal device. The terminal device sends an operation instruction to the video rendering device. The video rendering device generates the first video according to the operation instruction, generates the second video according to the first video, and sends the second video to the terminal device. The terminal device is used to display the second video and generate an operation instruction. It should be understood that in the description of Figure 1 above, the video rendering device generates the first video according to the request of the terminal device in the end-cloud collaboration. At this time, the request of the terminal device carries the operation instruction.

[0097] As described above, interpolated images are obtained by the video rendering device based on the original image. The quality of the interpolated images will affect the quality of the second video. To improve the quality of the interpolated images, the video rendering device can obtain the interpolated images based on an operation instruction and the original image. For example, the video rendering device receives an operation instruction for the second video from a terminal device. Based on the operation instruction, the video rendering device performs optical flow estimation on the Nth frame image and the N+1th frame image to obtain a first optical flow. The video rendering device then performs an affine transformation on the N+1th frame image and the first optical flow to obtain a first interpolated image.

[0098] According to the previous description, optical flow estimation is used to predict the motion information of objects in an image to determine the position of the object at a certain moment in the future. Operation instructions are used to control objects in the image. Therefore, operation instructions are conducive to improving the accuracy of optical flow estimation, thereby improving the quality of interpolated images. For example, in the example of Figure 8a, operation instruction 1 instructs the person in the image to move to the left at a first speed. The position of the person in the optical flow estimation image in the next frame image (that is, the optical flow image corresponding to the first interpolated image) is located at point a. The video rendering device can correct the position of point a by comparing with the "first speed" and "left" as a reference, thereby improving the quality of the interpolated image. For another example, the video rendering device directly uses "first speed" and "left" as variable parameters in the optical flow estimation to obtain a high-quality optical flow image.

[0099] It should be understood that the above description of the method for obtaining an interpolated image based on the first instruction is illustrative. In actual applications, those skilled in the art may obtain an interpolated image based on the first instruction using other methods. For example, after obtaining interpolated image n+1 using the method shown in FIG. 7a , interpolated image n+1 is modified using a neural network model to obtain a modified interpolated image. The input to the neural network model includes the first instruction and interpolated image n+1. The output of the neural network model includes the modified interpolated image. As described above, N is an integer greater than or equal to 1. When N is greater than 1, it indicates that the first video includes images before the N+0th frame image. For example, the first video also includes the N-1th frame image. In this case, similar to the description of obtaining the first interpolated image using the video rendering device, the video rendering device can obtain a second interpolated image based on the N-1th frame image and the Nth frame image. The second interpolated image is located between the Nth frame image and the N+1th frame image in the second video. The operation instruction received by the video rendering device from the terminal device includes the target instruction generated by the terminal device when displaying the second interpolated image and the N+1th frame image. To improve the quality of the interpolated image, the video rendering device may generate a first interpolated image based on the target instruction, the Nth frame image, and the N+1th frame image. For example, the video rendering device may perform optical flow estimation on the Nth frame image and the N+1th frame image based on the target instruction to obtain a first optical flow. The video rendering device may then perform an affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated image.

[0100] Figure 9 is a diagram illustrating the correspondence between the first video, the second video, and the operation instruction stream provided in an embodiment of the present application. As shown in Figure 9, the first video includes the N-1th frame image, the N+0th frame image, and the N+1th frame image. The second video includes the N-1th frame image, the N+0th frame image, the n+0th frame image, the N+1th frame image, and the n+1th frame image. The N-1th frame image, the N+0th frame image, and the N+1th frame image are original images. The n+0th frame image and the n+1th frame image are interpolated frames. The operation instruction stream includes operation instructions 1 to 5. Operation instructions 1 to 5 correspond one-to-one to the five images in the second video. The n+0th frame image is also called the second interpolated frame image. The n+0th frame image is obtained based on the N-1th frame image and the N+0th frame image. For the description of obtaining the second interpolated frame image, please refer to the description of obtaining the first interpolated frame image in this application. The n+1th frame image is also called the first interpolated frame image. The n+1th frame image is obtained based on the N+0th frame image, the N+1th frame image, and the target instruction. The target instruction includes operation instruction 3 and operation instruction 4. Operation instruction 3 is the operation instruction generated by the terminal device when displaying the second interpolated frame image. Operation instruction 4 is the operation instruction generated by the terminal device when displaying the N+1th frame image.

[0101] The present application also provides a video acquisition method. The video rendering method is applied to a terminal device. The terminal device can be a terminal device in end-cloud collaboration, such as a PC, mobile phone, or tablet. Figure 10 is a flow chart of the video acquisition method provided in an embodiment of the present application. As shown in Figure 10, the video acquisition method includes the following steps.

[0102] In step 1001, the terminal device receives a second video from a video rendering device. The second video is obtained by interpolating the first video by the video rendering device. The first video includes the Nth frame image and the N+1th frame image. The second video includes the Nth frame image, the N+1th frame image and the first interpolated frame image. The first interpolated frame image is the image after the N+1th frame image in the second video.

[0103] In step 1002, the terminal device displays the second video.

[0104] It should be understood that the description of the video acquisition method shares similarities with the description of the video rendering method in Figures 1 to 9 . Therefore, the description of the video acquisition method in Figure 10 can refer to the description of the video rendering method in Figures 1 to 9 . For example, the first interpolated frame image is obtained based on the Nth frame image and the N+1th frame image. In another example, the first interpolated frame image is the next frame image after the N+1th frame image in the second video. In another example, the video acquisition method further includes the following steps: the terminal device sends an operation instruction for the second video to the video rendering device. The video rendering device is configured to obtain the first interpolated frame image based on the operation instruction, the Nth frame image, and the N+1th frame image. In another example, before displaying the second video, the video acquisition method further includes the following steps: the terminal device decodes the encoded second video. The terminal device displays the decoded second video. The meanings of encoding and decoding are explained below.

[0105] Encoding refers to the process of converting an original video file into another format through compression technology. Directly transmitting the raw video image data requires significant bandwidth resources. To conserve bandwidth, the encoder compresses the game image before transmission, reducing the image data size and converting it into a unified encoding format. Decoding refers to the process of using a specific algorithm to decompress the encoded game image, or converting the electrical pulse signal into the information or data it represents.

[0106] Figure 11 is a first structural diagram of a video rendering device provided by an embodiment of the present application. As shown in Figure 11, the video rendering device 1100 includes a first acquisition unit 1101, a second acquisition unit 1103 and an interpolation unit 1102. The first acquisition unit 1101 is used to acquire a first video. The first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second acquisition unit 1103 is used to acquire a first instruction. The first instruction instructs to operate on the second video, and the second video is obtained after interpolation of the first video. The interpolation unit 1102 is used to obtain a first interpolation image based on the first instruction, the Nth frame image and the N+1th frame image. Among them, the second video includes the Nth frame image, the N+1th frame image and the first interpolation image. The first interpolation image is an image after the N+1th frame image in the second video.

[0107] It should be understood that the description of the video rendering device 1100 in FIG11 is similar to the description of the video rendering method in the aforementioned FIG1 to FIG9 . Therefore, for the description of the video rendering device 1100 in FIG11 , reference can be made to the description of the video rendering method in the aforementioned FIG1 to FIG9 . For example, the first interpolated frame image is obtained based on the Nth frame image and the N+1th frame image. For another example, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. For another example, the video rendering device 1100 also includes a receiving unit. The receiving unit is used to receive an operation instruction for the second video from the terminal device. The video rendering device is used to obtain the first interpolated frame image based on the operation instruction, the Nth frame image and the N+1th frame image.

[0108] FIG12 is a second structural diagram of the video rendering device provided in an embodiment of the present application. As shown in FIG12 , based on FIG11 , the video rendering device 1100 further includes a receiving unit 1201 and a sending unit 1202. The receiving unit 1201 is used to receive a request for generating a first video from a terminal device, the request including an operation instruction. The acquisition unit 1101 includes an instruction decoding unit and a video generation unit. The instruction decoding unit is used to decode the operation instruction to obtain a decoded instruction. The video generation unit is used to generate the first video according to the decoded instruction. The interpolation unit 1102 includes a super-resolution processing unit, an optical flow processing unit, an affine transformation unit, a video synthesis unit, and an encoding unit. The super-resolution processing unit is used to perform super-resolution processing on the original image in the first video to obtain a super-resolution image. The optical flow processing unit is used to perform optical flow estimation on the super-resolution image to obtain an optical flow image. The affine transformation unit is used to perform affine transformation on the optical flow image and the super-resolution image to obtain an interpolation image. The video synthesis unit is used to synthesize the interpolation image and the image in the first video to obtain a second video. The encoding unit is configured to encode the second video to obtain the encoded second video. The sending unit 1202 is configured to send the encoded second video to the terminal device.

[0109] Figure 13 is a third structural diagram of the video rendering device provided in an embodiment of the present application. As shown in Figure 13, the video rendering device 1300 includes a transceiver 1301 and a processor 1302. The processor 1302 is used to obtain a first video, and the first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The processor 1302 is also used to obtain a first instruction, and the first instruction instructs to operate on the second video, and the second video is obtained after interpolation of the first video. The processor 1302 is also used to obtain a first interpolated image based on the first instruction, the Nth frame image, and the N+1th frame image. Among them, the second video includes the Nth frame image, the N+1th frame image, and the first interpolated image, and the first interpolated image is the image after the N+1th frame image in the second video. The transceiver 1301 is used to send the second video to the terminal device.

[0110] It should be understood that the description of the video rendering device 1300 in FIG13 is similar to the description of the video rendering method in the aforementioned FIG1 to FIG9 . Therefore, for the description of the video rendering device 1300 in FIG13 , reference may be made to the description of the video rendering method in the aforementioned FIG1 to FIG9 . For example, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. For another example, the processor 1302 is used to obtain the first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, including: the processor 1302 is used to perform optical flow estimation on the Nth frame image and the N+1th frame image based on the first instruction to obtain the first optical flow; the processor 1302 is used to perform affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated frame image.

[0111] FIG14 is a fourth structural diagram of a video rendering device provided in an embodiment of the present application. As shown in FIG14 , based on FIG13 , a transceiver 1301 includes a receiver and a transmitter. The receiver is configured to receive a request from a terminal device to generate a first video, the request including an operation instruction. The processor 1302 includes an instruction decoder, a graphics processing unit (GPU), a super-resolution processor, an optical flow engine, an affine transformation processor, a video synthesizer, and an encoder. The instruction decoder is configured to decode the operation instruction to obtain a decoded instruction. The GPU is configured to generate the first video based on the decoded instruction. The super-resolution processor is configured to super-resolution the original image in the first video to obtain a super-resolution image. The optical flow engine is configured to perform optical flow estimation on the super-resolution image to obtain an optical flow image. The affine transformation processor is configured to perform affine transformation on the optical flow image and the super-resolution image to obtain an interpolated image. The video synthesizer is configured to synthesize the interpolated image with the image in the first video to obtain a second video. The encoder is configured to encode the second video to obtain an encoded second video. The sending unit 1202 is configured to send the encoded second video to the terminal device.

[0112] Figure 15 is a first structural diagram of a terminal device provided in an embodiment of the present application. As shown in Figure 15, the terminal device 1500 includes a receiving unit 1501 and a display unit 1502. Among them, the receiving unit 1501 is used to receive a second video from a video rendering device. The second video is obtained by the video rendering device interpolating the first video, and the first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second video includes the Nth frame image, the N+1th frame image and the first interpolated image. The first interpolated image is obtained based on the first instruction, the Nth frame image and the N+1th frame image, and the first instruction indicates to operate on the second video. The first interpolated image is an image after the N+1th frame image in the second video. The display unit 1502 is used to display the second video.

[0113] It should be understood that the description of the terminal device 1500 in Figure 15 is similar to the description of the video rendering method in Figures 1 to 9 above. Therefore, for the description of the terminal device 1500 in Figure 15, reference can be made to the description of the video rendering method in Figures 1 to 9 above. For example, the first interpolated frame image is obtained based on the Nth frame image and the N+1th frame image. For another example, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. For another example, the terminal device also includes a sending unit. The sending unit is used to send an operation instruction for the second video to the video rendering device. The video rendering device is used to obtain the first interpolated frame image based on the operation instruction, the Nth frame image and the N+1th frame image.

[0114] Figure 16 is a second structural diagram of the terminal device provided in an embodiment of the present application. As shown in Figure 16, based on Figure 15, the terminal device 1500 also includes a decoding unit 1601, a touch response unit 1602 and a transmitting unit 1603. Among them, the receiving unit 1501 is used to receive the encoded second video from the video rendering device. The decoding unit 1601 is used to decode the encoded second video to obtain the decoded second video. The display unit 1502 is used to display the second video. The touch response unit 1602 is used to generate an operation instruction. The transmitting unit 1603 is used to send an operation instruction to the video rendering device.

[0115] Figure 17 is a third structural diagram of the terminal device provided in an embodiment of the present application. As shown in Figure 17, the terminal device 1700 includes a transceiver 1701 and a display 1702. The transceiver 1701 is used to receive a second video from a video rendering device. The second video is obtained by the video rendering device interpolating the first video. The first video includes the Nth frame image and the N+1th frame image, where N is an integer greater than 0. The second video includes the Nth frame image, the N+1th frame image, and the first interpolated frame image. The first interpolated frame image is obtained based on the first instruction, the Nth frame image, and the N+1th frame image, and the first instruction indicates to operate on the second video. The first interpolated frame image is an image after the N+1th frame image in the second video. The display 1702 is used to display the second video.

[0116] It should be understood that the description of the terminal device 1700 in FIG17 is similar to the description of the video rendering method in the aforementioned FIG1 to FIG9 . Therefore, the description of the terminal device 1700 in FIG17 can refer to the description of the video rendering method in the aforementioned FIG1 to FIG9 . For example, the first interpolated frame image is the next frame image of the N+1th frame image in the second video. For another example, the transceiver 1701 is also used to send an operation instruction for the second video to the video rendering device. The video rendering device is used to obtain the first interpolated frame image based on the operation instruction, the Nth frame image, and the N+1th frame image.

[0117] Figure 18 is a fourth structural diagram of a terminal device provided in an embodiment of the present application. As shown in Figure 18, based on Figure 17, the terminal device 1700 further includes a decoder 1801 and a touch screen 1802. The transceiver includes a receiver and a transmitter. Among them, the receiver is used to receive the encoded second video from the video rendering device. The decoder 1801 is used to decode the encoded second video to obtain the decoded second video. The display 1702 is used to display the second video. The touch screen 1802 is used to generate an operation instruction. The transmitter is used to send an operation instruction to the video rendering device.

[0118] Figure 19 is a schematic diagram of the structure of a video rendering system provided in an embodiment of the present application. As shown in Figure 19, the video rendering system includes a video rendering device 1901 and a terminal device 1902. For a description of video rendering device 1901, reference may be made to the description of any of Figures 11 to 14 above. For a description of terminal device 1902, reference may be made to the description of any of Figures 15 to 18 above. Video rendering device 1901 is used to obtain a first video. The first video includes an Nth frame image and an N+1th frame image, where N is an integer greater than 0. Video rendering device 1901 is also used to interpolate frames in the first video to obtain a second video. The second video includes an Nth frame image, an N+1th frame image, and a first interpolated frame image. The first interpolated frame image is obtained based on a first instruction, an Nth frame image, and an N+1th frame image, where the first instruction indicates an operation to be performed on the second video. The first interpolated frame image is the image after the N+1th frame image in the second video. Video rendering device 1901 is also used to send the second video to terminal device 1902. Terminal device 1902 is used to display the second video.

[0119] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A video rendering method, characterized in that: include: Acquire a first video, where the first video includes an Nth frame image and an N+1th frame image, where N is an integer greater than 0; Obtaining a first instruction, where the first instruction instructs to operate on a second video, where the second video is obtained by inserting frames into the first video; Based on the first instruction, the Nth frame image and the N+1th frame image, a first interpolated frame image is obtained, wherein the second video includes the Nth frame image, the N+1th frame image and the first interpolated frame image, and the first interpolated frame image is the image after the N+1th frame image in the second video.

2. The video rendering method according to claim 1, wherein: Obtaining the first interpolated image based on the first instruction, the Nth frame image, and the N+1th frame image includes: performing optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain a first optical flow.

3. The video rendering method according to claim 2, wherein: The first video further includes an N-1th frame image, the second video further includes a second interpolated frame image, the second interpolated frame image is obtained based on the N-1th frame image and the Nth frame image, and the second interpolated frame image is located between the Nth frame image and the N+1th frame image in the second video, and the first instruction includes a target instruction, and the target instruction instructs to operate on the second interpolated frame image and the N+1th frame image; The performing optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow includes: performing optical flow estimation on the Nth frame image and the N+1th frame image according to the target instruction to obtain the first optical flow.

4. The video rendering method according to claim 2 or 3, characterized in that: The obtaining of the first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image includes: performing affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated frame image.

5. The video rendering method according to claim 4, characterized in that: The method further comprises: Obtaining a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image; The performing optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow includes: performing optical flow estimation on a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image according to the first instruction to obtain the first optical flow; The method further comprises: Upsampling the first optical flow to obtain a second optical flow; Performing super-resolution on the downsampled image of the N+1th frame image to obtain a first image of the N+1th frame image; The performing an affine transformation on the (N+1)th frame image and the first optical flow to obtain the first interpolated frame image includes: performing an affine transformation on the first image and the second optical flow to obtain the first interpolated frame image.

6. The video rendering method according to claim 5, characterized in that: The method further comprises: Obtaining a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image; Performing super-resolution on the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image to obtain a first image of the Nth frame image and a second image of the N+1th frame image; The performing optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow includes: performing optical flow estimation on the first image and the second image according to the first instruction to obtain the first optical flow; The performing an affine transformation on the (N+1)th frame image and the first optical flow to obtain the first interpolated frame image includes: performing an affine transformation on the second image and the first optical flow to obtain the first interpolated frame image.

7. The video rendering method according to any one of claims 1 to 6, characterized in that: The first interpolated frame image is the next frame image of the N+1th frame image in the second video.

8. A video rendering device, characterized in that: Includes a transceiver and a processor, including: The processor is configured to acquire a first video, where the first video includes an Nth frame image and an N+1th frame image; The processor is further configured to obtain a first instruction, wherein the first instruction instructs to operate on a second video, where the second video is obtained by inserting frames into the first video; The processor is further configured to obtain a first interpolated frame image based on the first instruction, the Nth frame image, and the N+1th frame image, wherein the second video includes the Nth frame image, the N+1th frame image, and the first interpolated frame image, and the first interpolated frame image is an image after the N+1th frame image in the second video; The transceiver is used to send the second video to the terminal device.

9. The video rendering device according to claim 8, characterized in that The processor is used to obtain the first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, including: the processor is used to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow.

10. The video rendering device according to claim 9, characterized in that The first video further includes an N-1th frame image, the second video further includes a second interpolated frame image, the second interpolated frame image is obtained based on the N-1th frame image and the Nth frame image, and the second interpolated frame image is located between the Nth frame image and the N+1th frame image in the second video, and the first instruction includes a target instruction, and the target instruction instructs to operate on the second interpolated frame image and the N+1th frame image; The processor is used to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, including: the processor is used to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the target instruction to obtain the first optical flow.

11. The video rendering device according to claim 9 or 10, characterized in that: The processor is configured to obtain the first interpolated frame image based on the first instruction, the Nth frame image and the N+1th frame image, including: the processor is configured to perform affine transformation on the N+1th frame image and the first optical flow to obtain the first interpolated frame image.

12. The video rendering device according to claim 11, characterized in that The processor is further configured to obtain a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image; The processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, comprising: the processor is configured to perform optical flow estimation on a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image according to the first instruction to obtain the first optical flow; The processor is further configured to upsample the first optical flow to obtain a second optical flow; The processor is further configured to perform super-resolution on the downsampled image of the N+1th frame image to obtain a first image of the N+1th frame image; The processor is configured to perform an affine transformation on the (N+1)th frame image and the first optical flow to obtain the first interpolated frame image, including: the processor is configured to perform an affine transformation on the first image and the second optical flow to obtain the first interpolated frame image.

13. The video rendering device according to claim 11, characterized in that The processor is further configured to obtain a downsampled image of the Nth frame image and a downsampled image of the N+1th frame image; The processor is further configured to perform super-resolution on the downsampled image of the Nth frame image and the downsampled image of the N+1th frame image to obtain a first image of the Nth frame image and a second image of the N+1th frame image; The processor is configured to perform optical flow estimation on the Nth frame image and the N+1th frame image according to the first instruction to obtain the first optical flow, which includes: the processor is configured to perform optical flow estimation on the first image and the second image according to the first instruction to obtain the first optical flow; The processor is configured to perform an affine transformation on the (N+1)th frame image and the first optical flow to obtain the first interpolated frame image, including: the processor is configured to perform an affine transformation on the second image and the first optical flow to obtain the first interpolated frame image.

14. The video rendering device according to any one of claims 8 to 13, characterized in that: The first interpolated frame image is the next frame image of the N+1th frame image in the second video.

15. A video rendering device, characterized in that: It includes a first acquisition unit, a second acquisition unit and a frame insertion unit, wherein: The first acquisition unit is used to acquire a first video, where the first video includes an Nth frame image and an N+1th frame image; The second acquiring unit is configured to acquire a first instruction, where the first instruction instructs to operate on a second video, where the second video is obtained by inserting frames into the first video; The interpolation unit is used to obtain a first interpolation image based on the first instruction, the Nth frame image and the N+1th frame image, wherein the second video includes the Nth frame image, the N+1th frame image and the first interpolation image, and the first interpolation image is an image after the N+1th frame image in the second video.

16. A video acquisition method, characterized in that: include: receiving a second video from a video rendering device, where the second video is obtained by the video rendering device through interpolation of a first video, the first video including an Nth frame image and an N+1th frame image, the second video including the Nth frame image, the N+1th frame image, and a first interpolated frame image, the first interpolated frame image being an image subsequent to the N+1th frame image in the second video, the first interpolated frame image being obtained based on a first instruction, the Nth frame image, and the N+1th frame image, wherein the first instruction instructs an operation to be performed on the second video; The second video is displayed.

17. A terminal device, characterized in that: Includes a transceiver and a display, wherein: The transceiver is configured to receive a second video from a video rendering device, where the second video is obtained by the video rendering device through frame interpolation of a first video, the first video including an Nth frame image and an N+1th frame image, the second video including the Nth frame image, the N+1th frame image, and a first interpolated frame image, the first interpolated frame image being an image subsequent to the N+1th frame image in the second video, the first interpolated frame image being obtained based on a first instruction, the Nth frame image, and the N+1th frame image, and the first instruction instructing an operation to be performed on the second video; The display is used to display the second video.

18. A video rendering system, characterized in that: Includes video rendering equipment and terminal equipment, including: The video rendering device is used to obtain a first video, where the first video includes an Nth frame image and an N+1th frame image; The video rendering device is further configured to obtain a first instruction, wherein the first instruction instructs to operate on a second video, where the second video is obtained by inserting frames into the first video; The video rendering device is further configured to obtain a first interpolated frame image based on the first instruction, the Nth frame image, and the N+1th frame image, wherein the second video includes the Nth frame image, the N+1th frame image, and the first interpolated frame image, and the first interpolated frame image is an image subsequent to the N+1th frame image in the second video; The video rendering device is further configured to send the second video to the terminal device; The terminal device is used to display the second video.

Images