Method, medium, and computing device for video encoding

Abstract

The present disclosure provides a method, medium and computing device for video encoding, relating to the field of image and video processing, and particularly to the field of video coding and compression, and the implementation solution is as follows: receiving a sequence of image frames from a rendering engine; determining whether a current frame in the sequence of image frames is a rendered frame or an interpolated frame; in response to determining that the current frame is a rendered frame, encoding the current frame in a first encoding pipeline based on an encoding result of a previous rendered frame of the current frame in the sequence of image frames; and in response to determining that the current frame is an interpolated frame, encoding the current frame in a second encoding pipeline different from the first encoding pipeline based on encoding results of two previous rendered frames of the current frame in the sequence of image frames.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing, and more particularly to the field of video encoding, decoding and compression, and can be applied to scenarios such as cloud gaming and remote desktops. In particular, it relates to methods, media and computing devices for video encoding. Background Technology

[0002] Real-time interactive streaming, especially in applications like cloud gaming and remote desktops, presents stringent challenges to video transmission efficiency. In these applications, the core of user experience lies in achieving a balance between low latency and high image quality. On one hand, to provide a smooth, lag-free visual experience, video needs to be transmitted at a high frame rate (e.g., 60 frames per second); on the other hand, to cope with users' fluctuating and often limited network bandwidth, video data must be efficiently compressed to transmit at the lowest possible bitrate. There is an inherent contradiction between these two approaches.

[0003] To balance server rendering load and network bandwidth, the industry commonly employs frame interpolation technology. The basic idea is that the server generates and encodes "rendered frames" at a lower frame rate (e.g., 30 frames per second), while simultaneously generating new "interpolated frames" between these rendered frames using an algorithm, encoding them together, and transmitting them to the client. In this way, the client can ultimately play the video at a higher, smoother frame rate (e.g., 60 frames per second).

[0004] However, the introduction of interpolated frames also brings new coding efficiency and real-time performance issues. In existing technologies, interpolated frames are usually encoded as ordinary forward prediction frames (P-frames), meaning that the encoder recalculates the prediction information based on the reference frame through motion estimation and motion compensation processes. This approach has significant inefficiencies: the interpolated frame itself is explicitly generated by the algorithm, and its motion relationships and pixel changes are determined during the generation stage. However, the encoder still needs to repeatedly perform a computationally intensive motion search process to "rediscover" the known motion information, resulting in unnecessary computational redundancy. Furthermore, due to the incomplete matching of prediction with the interpolation generation process, it is difficult to further improve compression efficiency.

[0005] Furthermore, in existing encoding pipelines, rendered frames and interpolated frames are typically arranged in the final playback order and sequentially subjected to forward predictive coding within the same encoding pipeline. (Playback order index) For example, located in two rendering frames , Interpolation frames between This is typically achieved through an interpolation algorithm after the two rendering frames mentioned above have been generated. In this case, the encoder must wait during processing. Only after the generation and encoding are completed can the previously obtained data be obtained. Encoding introduces additional waiting time during the encoding stage, increasing overall encoding latency, reducing the real-time performance of the video stream, and consequently affecting the interactive experience.

[0006] To address the aforementioned issues, some improvements have been proposed in existing technologies. For example, in some solutions, the server only transmits lightweight auxiliary information (such as motion vectors or control parameters) to the client, with the client performing frame interpolation locally. However, this type of solution shifts the computational complexity to the client side, and the computing power and power consumption of client devices (such as mobile terminals and set-top boxes) are usually limited, making it difficult to stably support high-quality interpolation operations. Furthermore, the interpolation algorithms and implementation methods used by different client devices vary, making it difficult to maintain consistency in the final display effect, and the server cannot uniformly control the user experience.

[0007] Therefore, in the field of real-time interactive streaming media applications, there is still a pressing technical problem to be solved: how to effectively improve the compression efficiency of interpolated frames and reduce encoding latency within the existing video coding standard framework without significantly increasing the client's computational burden, thereby alleviating the bandwidth bottleneck and latency problems faced by high frame rate real-time video transmission. Summary of the Invention

[0008] In order to solve the above-mentioned technical problems, this disclosure provides a method, medium and computing device for video encoding, a method for video decoding and a method for storing bit streams.

[0009] According to a first aspect of this disclosure, a method for video encoding is provided, the method comprising: receiving an image frame sequence from a rendering engine; determining whether a current frame in the image frame sequence is a rendered frame or an interpolated frame; in response to determining that the current frame is a rendered frame, encoding the current frame in a first encoding pipeline based on the encoding result of a previous rendered frame in the image frame sequence; and in response to determining that the current frame is an interpolated frame, encoding the current frame in a second encoding pipeline different from the first encoding pipeline based on the encoding results of two previous rendered frames in the image frame sequence.

[0010] In some embodiments, the method further includes obtaining corresponding motion information generated during the calculation of one or more interpolated frames in an image frame sequence from a rendering engine, and further encoding the corresponding interpolated frames in a second encoding pipeline based on the corresponding motion information.

[0011] In some embodiments, the first coding pipeline is configured to perform coding based on forward prediction, and the second coding pipeline is configured to perform coding based on bidirectional prediction or based on forward prediction using at least two reference frames.

[0012] In some embodiments, the method further includes: in response to determining that the current frame is an interpolated frame, encoding the current frame in a second encoding pipeline different from the first encoding pipeline based on the encoding results of the second and third rendered frames preceding the current frame in the image frame sequence, wherein the first encoding pipeline is configured to encode based on forward prediction and the second encoding pipeline is configured to encode based on forward prediction using at least two reference frames.

[0013] In some embodiments, the method further includes: for each rendered frame in the image frame sequence, forming a micro-image group with the encoding result of the rendered frame and the encoding result of the interpolation frame between the rendered frame and the next rendered frame in the image frame sequence, for transmission in the same data packet.

[0014] In some embodiments, the first encoding pipeline and the second encoding pipeline are configured to encode using different video encoders.

[0015] According to a second aspect of this disclosure, a method for video decoding is provided, the method comprising: receiving a data packet generated by the method according to a first aspect of this disclosure; decoding the data packet to obtain a rendered frame and an interpolated frame; and reordering the rendered frame and the interpolated frame to obtain an image frame sequence.

[0016] According to a third aspect of this disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by a processor of a computer, cause the processor to perform the method described according to a first or second aspect of this disclosure.

[0017] According to a fourth aspect of this disclosure, a method for storing a bit stream is provided, comprising performing the method described according to a first aspect of this disclosure to generate a bit stream and storing the bit stream.

[0018] According to a fifth aspect of this disclosure, a computing device is provided, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the computing device to perform the method according to a first or second aspect of this disclosure.

[0019] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0020] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0021] Figure 1 This is a schematic diagram of the encoding method for interpolated frames in existing technologies.

[0022] Figure 2 This is a flowchart illustrating a video encoding method according to an exemplary embodiment.

[0023] Figure 3A , Figure 3B as well as Figure 4 This is a schematic diagram illustrating an encoding method according to an exemplary embodiment.

[0024] Figure 5 This is a flowchart illustrating a video decoding method according to an exemplary embodiment.

[0025] Figure 6 An example configuration of a computing device that can be used to implement the methods described herein is shown. Detailed Implementation

[0026] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0027] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0028] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0029] Figure 1 This is a schematic diagram of the encoding method for interpolated frames in existing technologies.

[0030] Figure 1This includes a rendering engine 110 and an encoder 120. The rendering engine 110 is responsible for generating image frame sequences 111. In real-time interactive streaming applications, the rendering engine is typically deployed on the server side to generate image frames in real time based on user input and scene status. For example, in cloud gaming or remote desktop scenarios, the rendering engine can perform geometric transformations, lighting calculations, texture mapping, and post-processing operations on a 3D scene based on a graphics processing unit (GPU), thereby generating one or more rendered images in each rendering cycle. During the rendering process, the rendering engine typically maintains the position, pose, and motion information of each object in the scene and outputs corresponding 2D image results based on changes in the camera's perspective. The generated image frame sequence 111 is then fed into the encoder 120 to form a video data stream, such as a bitstream, that can be transmitted over a network. The encoder 120 can operate according to proprietary or industry standards (e.g., Universal Video Codec (VVC), Joint Exploratory Test Model (JEM), High Efficiency Video Codec (HEVC / H.265), Advanced Video Codec (AVC / H.264), Moving Picture Experts Group (MPEG) codec) or extensions of such standards (e.g., encoding and decoding video data).

[0031] exist Figure 1 In the example, rendering engine 110 is configured to generate image frame sequence 111 at an interpolation rate of 1x, as shown by the subscript in the figure. This indicates the playback order of the image frames.

[0032] like Figure 1 As shown, at the initial moment, the rendering engine 110 generates the first image frame in the image frame sequence 111, i.e., the rendering frame, through rendering. This rendering frame It is then passed to encoder 120 and encoded in encoding pipeline 121.

[0033] The encoding pipeline 121 can be implemented using a typical architecture found in existing video coding standards, such as modules including a data memory, a prediction processing unit, a decoded image buffer (DPB), a residual calculation unit (e.g., a subtractor), a transform processing unit, a quantization processing unit, and an entropy coding unit. The prediction processing unit is typically configured to predict the current image to be encoded and generate a predicted image. It may include, but is not limited to, a motion estimation unit, a motion compensation unit, a segmentation unit, an intra-frame prediction processing unit, and an intra-frame block copy (IBC) unit to support inter-frame prediction, intra-frame prediction, and block-level prediction processing based on already encoded regions, respectively. For brevity, the above modules are not listed in the table. Figure 1 As shown in the image.

[0034] During the encoding process, encoder 120 can perform prediction processing on the current image frame based on one or more reference images from DPB, and transform, quantize and entropy encode the prediction residual to generate the corresponding compressed bit stream.

[0035] like Figure 1 As shown, upon receiving the rendering frame After that, the encoder... Intra-frame prediction processing is performed to generate the corresponding intra-coded frame (I-frame) encoding result. . It can be stored in the DPB (not shown in the figure) and used as a reference frame for subsequent image frames. Specifically, when encoding subsequent rendered or interpolated frames, the encoder can base it on... Perform inter-frame prediction processing to reduce redundant information and improve coding efficiency.

[0036] In generation Afterwards, rendering engine 110 continued rendering and generated a second rendering frame. The generated rendering frames It is then passed to encoder 120. It is worth noting that, because the image frame sequence 111 needs to... and Interpolated frames played between Encoder 120 will not be executed immediately as it has not yet been generated. Encoding operations.

[0037] In obtaining and Then, the rendering engine uses an interpolation method to obtain the interpolated frame located between the two. For example, a rendering engine can be based on rendering... and The motion information obtained during the process (e.g., the motion vector for each pixel) is used to perform interpolation, for example, by interpolation. .

[0038] For example, assuming that according to the motion vector, median coordinate pixels at exist The middle corresponds to the coordinates pixels at Then the rendering engine can middle pixels at Set as .

[0039] It should be understood that the above interpolation methods are merely illustrative examples, and interpolation processing can also be performed by an interpolation engine or other processing modules independent of the rendering engine. Furthermore, the description of interpolation processing in this disclosure is only for illustrating the generation process of interpolated frames and does not constitute a limitation on specific interpolation algorithms or implementation methods.

[0040] In interpolation frame After generation is complete, interpolated frames It is passed to encoder 120 and along with the rendering frame. They enter the encoding pipeline 121 together. Following the existing encoding process, encoder 120 will interpolate the frames. Treated as a normal prediction frame and based on the reference frame stored in the DPB (e.g. ) for interpolated frames Perform inter-frame predictive coding to obtain the corresponding forward predictive frame coding results. After completing the interpolation of the frames... After encoding, encoder 120 further... Encoding results For reference, the rendered frames are waiting in the queue. Perform inter-frame prediction to obtain the corresponding forward prediction frame coding results. .

[0041] Subsequently, the rendering engine 110 and encoder 120 continue to execute the above process in a loop, obtaining rendering frames in sequence. Interpolated frames Rendering frames and interpolated frames And obtain the encoding results corresponding to each image frame respectively. , , and The specific processing procedure and encoding method are the same as described above, and will not be repeated here.

[0042] Depend on Figure 1 The processing flow shown indicates that existing technologies still have several areas for improvement in real-time high frame rate video encoding scenarios. Figure 1 Rendering frames in For example, from the completion of the rendering frame The time from the start of rendering to the time when the rendering frame is obtained is calculated. Encoding results The minimum delay duration incurred up to the specified time. It can be expressed by the following formula:

[0043] ,

[0044] in, Indicates the interpolated frame The time consumed in performing interpolation calculations. Indicates the interpolated frame The time consumed in performing forward prediction, and, Indicates the rendering frame The time consumed in performing forward prediction. In other words, the time taken to complete a rendering frame. After rendering, at least a minimum latency period consisting of three different latency factors is required. Only encoder 120 can output and render frames. The corresponding bitstream is provided for the client to receive, decode, and reconstruct the rendered frame.

[0045] On the one hand, due to interpolation frames The playback order in image frame sequence 111 is located at the rendering frame. and Between these frames, and the encoder 120 needs to encode the image frames according to the playback order, in the interpolation frames... Before the encoder 120 has generated and passed the data, the encoder 120 cannot start rendering the frame. The encoding operation cannot be performed directly, and the interpolation process must be completed. Therefore, the time consumed by the interpolation calculation is... This creates an unavoidable serial wait delay, thereby reducing the real-time performance of the encoding link.

[0046] On the other hand, in the above encoding process, in order to obtain the forward prediction frame encoding result... Encoder 120 needs to process the interpolated frames. Motion estimation is performed to determine the corresponding motion vectors. This type of motion estimation typically involves large-scale or multi-scale matching calculations within the reference frame, resulting in high computational complexity. In other words, when the interpolated frame is treated as a regular prediction frame, the encoder actually needs to search and calculate the motion relationships that have already been implicitly determined by the interpolation algorithm again, resulting in significant redundant computational overhead, which manifests as considerable [cost / efficiency]. This repetitive motion estimation not only increases the computational burden on the encoding side, but also further amplifies the minimum latency. This is not conducive to the stringent low-latency requirements of real-time interactive streaming applications.

[0047] Furthermore, due to limitations in the search strategy, search range, and accuracy of the motion estimation algorithm within the encoder, the encoding process is based on interpolated frames. The obtained motion vectors may not perfectly match the actual or ideal motion relationships used in the interpolation generation stage. In some cases, the estimated motion vectors may contain biases or errors, leading to increased prediction residuals. This not only reduces coding efficiency but may also affect subsequent data generation based on... Rendering frames for prediction This negatively impacts the quality of the encoding, leading to error propagation.

[0048] To at least address the aforementioned problems, this disclosure provides a method for video encoding. The method provided by this disclosure eliminates the dependency of render frame encoding on interpolation frame encoding by introducing a parallel encoding pipeline, allowing the encoder to encode the render frame immediately after the rendering operation is completed, without waiting for interpolation to finish. In some embodiments, the scheme provided by this disclosure utilizes bidirectional prediction frame (B-frame) technology under the existing video coding standard framework to encode the interpolation frame. By using the render frame associated with the interpolation frame as a reference frame for bidirectional prediction, the prediction residual of the interpolation frame can be reduced, thereby improving the encoding compression ratio and reducing the latency generated in input / output (I / O). In some embodiments, the scheme provided by this disclosure avoids searching and calculating motion relationships again in the encoding process by reusing motion information obtained in the interpolation calculation in the encoding of the interpolation frame, thereby significantly reducing the computational overhead and latency of the interpolation frame encoding, and further improving the accuracy of inter-frame prediction during the encoding process. The technical solutions and embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0049] Figure 2 This is a flowchart illustrating a video encoding method according to an exemplary embodiment of the present disclosure.

[0050] like Figure 2 As shown, video encoding method 200 includes operations 202, 204, 206, and 208. It can be executed using electronic devices deployed on a server. Figure 2 The video encoding method shown in the figure. The server-side electronic device typically has graphics rendering, video encoding, and data communication capabilities, and exemplarily includes a cloud server or edge server for cloud gaming or remote desktop services, which may integrate a central processing unit (CPU), a graphics processing unit (GPU), a hardware video encoder, and a network interface module.

[0051] In operation 202, a sequence of image frames is received from the rendering engine. The rendering engine can be any image generation module or processing unit used to generate or provide image data, which can produce image frames based on 3D rendering, graphics compositing, desktop capture, application screen output, or other image generation methods. As an example, the rendering engine can run on the graphics processing unit (GPU) of a server device, or in a GPU cluster in a cloud service environment, for example, by executing graphics rendering tasks in parallel using multiple GPUs. In some implementations, the rendering engine can also run in a virtualized environment, such as a game engine or graphics application instance based on a virtual GPU (vGPU).

[0052] In some embodiments, the rendering engine may be a generation module or processing unit with its own interpolation function. In some embodiments, the generation of an image frame sequence may be completed collaboratively by the rendering engine and a separate interpolation module, wherein the interpolation module is used to generate one or more interpolated frames between adjacent rendering frames. For the purpose of brevity, in some embodiments or drawings of this disclosure, the rendering engine and the interpolation module are collectively referred to as the "rendering engine," but this term does not mean that the rendering engine only includes the module for performing rendering operations.

[0053] It should be understood that, without departing from the technical solutions disclosed herein, a rendering engine may include a rendering module, an interpolation module, and other processing modules related to image frame generation, or may only include some of these modules. The above general terms are for illustrative purposes only and do not constitute a limitation on the functional composition or module boundaries of the rendering engine.

[0054] As used in this disclosure, an image frame sequence may include multiple image frames output continuously or discontinuously by a rendering engine. The image frames may be original rendered frames, processed intermediate frames, or interpolated frames, and may also include image frames in any order or combination. Each image frame may be represented as a two-dimensional pixel array, image block data, texture data, or other data forms capable of representing image content, and its resolution, color format, and frame rate are not limited.

[0055] In operation 204, it is determined whether the current frame in the image frame sequence is a rendered frame or an interpolated frame. The determination of the image frame type can be based on type information, generation method information, or context information associated with the current frame. For example, it can be determined based on control signaling, metadata, flag bits, or indication information provided by upstream modules. In some examples, the determination of the image frame type can also be based on the temporal position or sequence relationship of the image frame in the sequence, such as the insertion position of the current frame relative to adjacent frames, or based on predefined frame generation modes, frame rate conversion strategies, or scheduling rules.

[0056] In operation 206, in response to determining that the current frame is a rendered frame, the current frame is encoded in the first encoding pipeline based on the encoding result of the previous rendered frame in the image frame sequence. In other words, in the scheme provided in this application, the encoding of the rendered frame is based on the encoding result of a previously obtained rendered frame, without depending on the encoding result of the interpolated frame. In this way, the encoder does not have to wait for the rendering engine to complete the calculation of the interpolated frame before it can begin encoding the rendered frame, thereby significantly reducing encoding latency.

[0057] In some embodiments, the first coding pipeline can be configured to perform encoding based on forward prediction. In some examples, the forward prediction in the first coding pipeline can be based on one or more reference frames in the image frame sequence that precede the current frame. For example, the first coding pipeline can be configured to perform forward prediction of the current frame based on the encoding results of multiple previous rendered frames in the image frame sequence, or it can be configured to perform forward prediction at least in part based on other information—such as the encoding results of interpolated frames generated in the second coding pipeline described below. For example, the first coding pipeline can be configured to select the optimal prediction result from prediction results based on different reference frames; or the first coding pipeline can be configured to combine prediction results from different reference frames, for example, by weighting, selecting, or fusing prediction results obtained based on different previous rendered frames as reference frames, to reduce residuals and improve compression efficiency. Forward prediction can be performed at the frame level, block level, or sub-block level, or it can be implemented in combination with other motion compensation, intra-frame prediction, and inter-frame prediction algorithms. In some examples, the first encoding pipeline can also be configured to perform forward prediction based on different reference frame selection strategies, search range settings, or prediction accuracy configurations.

[0058] In operation 208, in response to determining that the current frame is an interpolated frame, the current frame is encoded in a second encoding pipeline, different from the first encoding pipeline, based on the encoding results of the two preceding rendered frames in the image frame sequence. Since the encoding of the interpolated frame is based on the encoding results of the two preceding rendered frames in the image frame sequence, compared to encoding the interpolated frame as a regular forward prediction frame, it provides more prediction reference sources for the interpolated frame, thereby improving prediction accuracy and encoding robustness, and further enhancing encoding compression efficiency.

[0059] In some embodiments, in response to determining that the current frame is an interpolated frame, the current frame can be encoded in a second encoding pipeline different from the first encoding pipeline, based on the encoding results of the second and third rendered frames preceding the current frame in the image frame sequence. This helps decouple the encoding of the interpolated frame from the encoding of adjacent rendered frames, thereby further eliminating the significant delay caused by waiting for forward prediction in the encoding of adjacent rendered frames during the interpolated frame encoding process. This technical advantage will be further explained below.

[0060] It should be understood that the first encoding pipeline and the second encoding pipeline can be implemented in a logically distinct manner. They can share some hardware resources or software modules, or they can be executed by different processing units or threads. The above description of the interpolation frame encoding method is for illustrative purposes only and does not constitute a limitation on the encoding pipeline structure or specific implementation method.

[0061] In some embodiments, the second encoding pipeline can be configured to encode based on bidirectional prediction. In some examples, bidirectional prediction in the second encoding pipeline can be based simultaneously on reference frames that precede the current frame in playback order and / or reference frames that follow the current frame in playback order. For example, the second encoding pipeline can be configured to perform bidirectional prediction on the current frame based on the encoding results of multiple previous rendered or interpolated frames in the image frame sequence. For example, the second encoding pipeline can be configured to select a prediction result from multiple prediction results obtained based on different combinations of reference frames, or to combine prediction results from different reference frames, such as weighting, selecting, or fusing multiple prediction results to reduce prediction residuals and improve compression efficiency. Similarly, the bidirectional prediction can be performed at the frame level, block level, or sub-block level, and can be combined with motion compensation, intra-frame prediction, inter-frame prediction, or other prediction mechanisms. Bidirectional prediction can be performed at the frame level, block level, or sub-block level, and can also be combined with other motion compensation, intra-frame prediction, and inter-frame prediction algorithms. In some examples, the second encoding pipeline can also be configured to perform bidirectional prediction based on different reference frame selection strategies, search range settings, or prediction accuracy configurations.

[0062] Generally speaking, it is widely believed in the field that although bidirectional prediction technology can help further improve coding quality, including compression rate, it usually introduces additional coding latency because it needs to refer to future frames and involves frame reordering. Therefore, it is not suitable for real-time interactive streaming scenarios that are highly sensitive to latency, such as cloud gaming and remote desktop.

[0063] However, as recognized in this disclosure, by introducing a parallel coding pipeline to decouple the rendering frame coding from the coding of previous interpolated frames, the additional latency introduced by bidirectional prediction can be effectively hidden by utilizing the time required for the interpolation computation itself. Thus, not only can bidirectional prediction improve coding efficiency, but it can also reduce overall coding latency, making bidirectional prediction technology suitable for real-time interactive streaming scenarios.

[0064] In some embodiments, the second coding pipeline can be configured to encode based on forward prediction using at least two reference frames. For example, encoding can be performed using unidirectional B-frames, which are defined in modern video coding standards such as HEVC / H.265. Unidirectional B-frames are a technique for encoding the current frame based on multiple reference frames that precede the current frame in playback order.

[0065] On the one hand, configuring the second coding pipeline to encode based on forward prediction using at least two reference frames—such as the coding method for unidirectional prediction B-frames described above—can improve the applicability of the coding method. For example, this allows the coding method to improve compression efficiency and reduce coding latency for interpolated frames generated based on extrapolation algorithms without changing the coding framework.

[0066] More importantly, by configuring the second encoding pipeline to encode based on forward prediction using at least two reference frames, it also helps to decouple the encoding of interpolated frames from the encoding of adjacent rendered frames, thereby further eliminating the significant latency caused by waiting for forward prediction in the encoding of adjacent rendered frames during the encoding of interpolated frames. This technical advantage will be further explained below.

[0067] In some embodiments, the video encoding method 200 further includes obtaining corresponding motion information generated during the calculation of one or more interpolated frames in the image frame sequence from the rendering engine, and further encoding the corresponding interpolated frames in a second encoding pipeline based on the corresponding motion information. In some examples, obtaining the corresponding motion information generated during the calculation of one or more interpolated frames in the image frame sequence from the rendering engine includes obtaining the corresponding motion information generated during the calculation of each interpolated frame in the image frame sequence from the rendering engine.

[0068] By directly utilizing the motion information generated during the interpolation process as the predicted or final value of the motion vector, the technical solution disclosed herein avoids repeatedly performing the traditional motion estimation search process on the interpolated frame during the encoding stage, thereby significantly reducing the computational overhead and time cost required for interpolated frame encoding. In particular, since the motion information is highly consistent with the generation process of the interpolated frame, the predicted image generated based on this motion information has high prediction accuracy, resulting in a significant reduction in the residual energy between the interpolated frame and its predicted image. Based on this, after transformation, quantization, and entropy coding, only a small number of bits are needed to represent and transmit the interpolated frame.

[0069] Furthermore, by directly using the motion information generated during the interpolation process, prediction bias introduced by re-estimating motion vectors during the coding stage can be avoided, thereby reducing the risk of abnormally increasing prediction residuals. Therefore, the technical solution disclosed herein not only improves the coding efficiency of the interpolated frame itself but also reduces the risk of error propagation in the coding link.

[0070] In some examples, encoding the corresponding interpolated frame in the second coding pipeline based on the motion information obtained during the interpolation process may include the following operations:

[0071] 1) Collect multiple motion vectors corresponding to the current coded block in the interpolated frame;

[0072] 2) Perform statistical processing on these motion vectors to obtain their average values;

[0073] 3) Remove motion vectors whose deviation from the average value exceeds a predetermined range;

[0074] 4) Average the remaining motion vectors and use the result as the motion compensation reference vector for the current coding block.

[0075] As can be seen, compared with traditional motion estimation methods that require matching search to be performed within a large search area, the above example directly uses the motion vector field generated during the interpolation process for motion compensation, avoiding the complex search process and ensuring that the computational complexity of motion compensation does not increase with the expansion of the search area.

[0076] In some embodiments, the video encoding method 200 further includes, for each rendered frame in the image frame sequence, forming a micro-image group with the encoding results of the rendered frames between that rendered frame and the next rendered frame in the image frame sequence, for transmission within the same data packet. By transmitting the encoding results of the interpolated frames and the rendered frames with which they are related in the same data packet, the efficiency of mechanisms such as packet encapsulation, buffering, packet loss recovery, and retransmission can be improved, and decoding wait time can be reduced, which is beneficial to improving the stability and latency performance of real-time playback.

[0077] In some embodiments, the first encoding pipeline and the second encoding pipeline belong to different video encoders. This approach improves the parallelism of encoding rendering frames and interpolated frames, reduces encoding resource contention, thereby lowering encoding latency and improving overall encoding throughput and real-time processing capabilities.

[0078] The following is combined with Figure 3A , Figure 3B The following will further elaborate on the relevant embodiments of the video encoding method provided in this disclosure.

[0079] Figure 3A This is a schematic diagram illustrating an encoding method according to an exemplary embodiment of the present disclosure.

[0080] Figure 3A This includes a rendering engine 310 and an encoder 320. The rendering engine 310 is configured to generate an image frame sequence 311 at an interpolation rate of 1x, meaning there is one interpolated frame between every two rendered frames. The encoder 320 is configured with two encoding pipelines—a first encoding pipeline 321 and a second encoding pipeline 322—and is configured to encode the input image frame sequence 311 to form a video data stream that can be transmitted over a network. Figure 3A And in the following figures of this disclosure, the rendering frame Interpolated frames Encoding results , All are related to Figure 1 The description has the same meaning, and the first render frame is rendered and encoded. The operation is also with Figure 1 The same applies, and for the sake of brevity, will not be repeated. It should be understood that the technical solutions disclosed herein are not limited to those described above. Figure 3A The interpolation scheme shown is not applicable to all interpolation frame techniques, including interpolation and extrapolation. Furthermore, the numbering of image frame playback order in this disclosure is for illustrative purposes only and does not imply any limitation on the actual position of the image frames in a real image frame sequence. For example, rendering frames... It can be the first rendered frame of a sequence of image frames, or the first rendered frame of one of the subsequences of a sequence of image frames that includes multiple subsequences.

[0081] like Figure 3A As shown, after the rendering frame is completed After rendering, render frames The first encoding pipeline 321 of the encoder 320 is passed to the render frame. Encoding results Forward predictive encoding is performed for reference. Simultaneously, the rendering engine 310 performs encoding based on the rendered frames. and rendering frames To generate interpolated frames As can be seen, with Figure 1 The prior art shown is different because in Figure 3A In the embodiments shown The encoding does not depend on interpolated frames that have not yet been generated. ,right Encoding and interpolation frames The calculations can be performed in parallel.

[0082] After obtaining the interpolated frame Then, the interpolated frame and the motion information generated during the calculation process are fed into the second encoding pipeline 322 of the encoder 320. The first encoding pipeline 321 has already completed the rendering of the frame. Encode and obtain the corresponding encoding results. In the case of the second encoding pipeline 322, the encoder 320 renders frames. Encoding results as well as Encoding results For reference—and based on the input motion information—the interpolated frame Perform bidirectional predictive coding to obtain the coding result. .

[0083] Subsequently, the rendering engine 310 and encoder 320 continue to execute the above process in a loop, obtaining rendering frames in sequence. Interpolated frames Rendering frames and interpolated frames And obtain the encoding results corresponding to each image frame respectively. , , and The specific processing procedure and encoding method are the same as described above, and will not be repeated here.

[0084] During the above encoding process, encoder 320 can render the frame generated in the first encoding pipeline 321. Encoding results Interpolated frames generated in the second encoding pipeline 322 Encoding results Composition about the first Micro-image group of image frames And transmit it in a single data packet. Similarly, encoder 320 can... Encoding results With interpolated frames Encoding results Composition about the first Micro-image group of image frames And transmitted in a single data packet. When the decoder (not shown in the figure) receives the micro-image group... After the data packet is received, it can be based on the encoding result. Decode the rendering frame And can be based on the previously received micro-image group Encoding results in With micro-image group Encoding results in Both are used to decode the interpolated frame. Furthermore, the decoder can interpolate frames. With rendering frames The frames are sorted according to the playback order and added to the output image frame sequence. In some embodiments, the decoder may have two or more decoding pipelines to decode the rendered frames and interpolated frames in parallel, thereby further reducing decoding latency.

[0085] like Figure 3A As shown, from the completed rendering frame The time from the rendering moment to obtaining the interpolated frame is calculated. Encoding results Up to that point, it is used to prepare for playback up to the rendered frame. Minimum latency required for encoded data It can be expressed by the following formula:

[0086] ,

[0087] in, Indicates the interpolated frame The time consumed in performing interpolation calculations. Indicates the interpolated frame The time consumed in performing bidirectional prediction, and, Indicates the rendering frame The time consumed by forward prediction. It can be seen that simply by changing the encoding and interpolation calculations from a serial relationship in traditional schemes (its minimum latency)... for The relationship has been changed to parallelism, and the minimum delay time in this embodiment is... The number of components was reduced from three to two, thereby reducing coding latency.

[0088] More importantly, since bidirectional prediction reuses the motion information already generated in the interpolation calculation in this embodiment, it avoids the delay caused by the most computationally expensive and time-consuming motion estimation process in video coding, thus preventing delays in the interpolation frame coding itself. Compared to the latency introduced by forward prediction of interpolated frames in traditional methods, this approach can overcome the limitations of traditional methods. A decrease of 50% to 90%, that is This further reduces encoding latency.

[0089] Figure 3B This is a schematic diagram illustrating an encoding method according to another exemplary embodiment. Compared to Figure 3A , Figure 3B The difference lies in the fact that there are two interpolation frames between two rendering frames, meaning the interpolation ratio is 2. For simplicity, Figure 3B Zhongyu Figure 3A The same parts will not be repeated.

[0090] like Figure 3B As shown, in this embodiment, encoder 320 is based on Encoding results and Encoding results To render frames in an image frame sequence The next two interpolation frames , (The playback positions of these two interpolated frames are located within the rendering frames) and rendering frames Bidirectional predictive coding is performed between (the two sides) to obtain the corresponding coding results. and and will , and Composition about the first Micro-image group of image frames Transmission can be performed in a single data packet. It should be understood that the technical solution disclosed herein is not limited to a specific interpolation ratio, but is widely applicable to various interpolation ratios and interpolation accuracies.

[0091] In some embodiments, multiple second encoding pipelines may exist to further improve encoding efficiency. For example, for an interpolation ratio of 2, two second encoding pipelines may be allocated to encode the two interpolated frames between the two rendered frames separately.

[0092] The following is combined with Figure 4 This will further elaborate on the video coding scheme provided in this disclosure.

[0093] Figure 4 This is a schematic diagram illustrating an encoding method according to yet another exemplary embodiment of the present disclosure.

[0094] Figure 4 It includes a rendering engine 410 and an encoder 420. The rendering engine 410 is configured to generate an image frame sequence 411 at an interpolation rate of 1. The encoder 420 is configured with a first encoding pipeline 421 and a second encoding pipeline 422, and is configured to encode the input image frame sequence 411 to form a video data stream that can be transmitted over a network. Figure 4 In the accompanying drawings, the meanings of the reference numerals are the same as those in the attached drawings. Figure 1 and Figure 3A , Figure 3B The corresponding items in the text have the same or similar meanings, and will not be repeated here.

[0095] Figure 4 The illustrated embodiments are compared to Figure 3A , Figure 3B The variation in the illustrated embodiment is that, Figure 4 In this process, the encoding of the interpolated frame completely avoids reference to future frames, instead relying on the encoding results of the second and third rendered frames preceding the interpolated frame in image frame sequence 411. The rightmost interpolated frame in the figure is used as an example. For example, in rendering engine 410, by analyzing rendering frames... and rendering frames Perform interpolation to generate interpolated frames Subsequently, encoder 420 bases its data on the interpolated frames in image frame sequence 411. Rendering frames that are not directly adjacent and rendering frames Encoding results and To obtain interpolated frames Encoding results In other words, encoder 420 backtracks through the interpolated frames. The B-frame is predicted unidirectionally, referencing the previous rendered frame of the next nearest neighbor. and the previous rendered frame of the next nearest neighbor Encode it. Similarly, for interpolated frames... The encoder 420 is based on the interpolated frames in the image frame sequence 411. The second rendering frame that is traced back. and the third rendering frame To interpolate frames Encode it. In this way, Figure 4 The implementation examples can further reduce encoding latency.

[0096] review Figure 3A Minimum delay time (in) As can be seen, besides the time consumed by encoding the interpolation frame itself... In addition, for interpolated frames Time consumed in interpolation calculation The time consumed by forward prediction of the nearest neighbor previously rendered frames. The higher of the two values ​​determines the overall latency. In scenarios such as cloud gaming, since the rendering engine itself can provide the motion information of the rendered frames, the time consumed by interpolation calculations is relatively small. It may be significantly less time than that consumed by forward predictive coding. At this point, the minimum delay duration It can be represented as + .

[0097] exist Figure 4 In the illustrated embodiment, by encoding the interpolated frame as a unidirectional predicted B-frame referencing the previous rendered frames of the next nearest and next-nearest neighbors, the encoding of the interpolated frame can be further avoided from waiting for the encoding result of the previous rendered frame of the nearest neighbor, thus minimizing the delay time. Controlled + This further reduces encoding latency.

[0098] Furthermore, in some embodiments, the encoder 420 can utilize existing motion information to synthesize a high-precision initial motion field estimate. Figure 4 For example, when performing interpolation frames During encoding, the encoder can obtain the data from the rendering engine 410 from the rendered frames. to interpolation frame The motion information is compared with the previously obtained rendering frames. to render frame The motion information is synthesized. This synthesis of motion information can be used to render frames. Encoding results to interpolation frame Encoding results The motion provides an extremely accurate initial prediction, thus greatly reducing the complexity of subsequent corrections or searches.

[0099] Overall, although the compression efficiency of unidirectional B-frame prediction may theoretically be slightly lower than that of bidirectional B-frame prediction due to the lack of future reference information, unidirectional B-frame prediction still utilizes the more flexible and efficient prediction modes supported by the B-frame data structure than P-frames. Through the motion field synthesis techniques described above, its prediction accuracy remains at a high level, thus still achieving a significant improvement in compression ratio compared to P-frames. This provides a different, more lax, trade-off between maximizing compression efficiency and minimizing processing latency, making it particularly suitable for real-time applications with the most stringent latency requirements.

[0100] According to one aspect of this disclosure, a method for video decoding is provided.

[0101] Figure 5 This is a flowchart illustrating a video decoding method according to an exemplary embodiment of the present disclosure.

[0102] like Figure 5 As shown, the video decoding method 500 includes operations 502, 504, and 506. In some examples, electronic devices may be used to perform these operations. Figure 5 The video decoding method shown is illustrated. Electronic devices typically possess video decoding, image display, and data communication capabilities, and exemplary examples include user terminal devices for cloud gaming or remote desktop services, such as personal computers, laptops, tablet computing devices, smartphones, set-top boxes, smart TVs, virtual reality / augmented reality terminals, or in-vehicle infotainment systems, or other terminal devices with display capabilities. Terminal devices may integrate a central processing unit (CPU), a graphics processing unit (GPU), a hardware video decoder, a display interface, and a network interface module.

[0103] In operation 502, a data packet is received. In some examples, the data packet may be generated according to embodiments described in this disclosure.

[0104] In Operation 504, the data packet is decoded to obtain rendered and interpolated frames. In some examples, the data packet can indicate whether each of the one or more image frames it contains is a rendered or interpolated frame through frame type identifiers, flags, or metadata. For example, a frame type field can be set in the data packet header or bitstream syntax elements to distinguish between directly rendered and interpolated image frames; alternatively, the frame type can be implicitly indicated by the frame's position in the sequence, a predefined frame mode, or decoding order information.

[0105] In addition, data packets may also indicate other common information related to image frame decoding, such as reference frame relationships, timestamps or display order information, encoding parameter sets, resolution and color format information, prediction mode information, quantization parameters, and error checking or synchronization information. It should be understood that the above indications are merely examples and do not constitute a limitation on the data packet structure or the information it contains.

[0106] In operation 506, the rendered frames and interpolated frames are reordered to obtain a sequence of image frames. In some examples, the reordering can be based on the frame playback order identifier, timestamp information, or display order indication information; for example, reordering rendered frames and interpolated frames received in encoded order into a sequence of image frames that conforms to the expected playback timing. In some implementations, the reordering can also be performed in conjunction with frame type information, inter-frame dependencies, or buffer state to ensure that interpolated frames are inserted between corresponding adjacent rendered frames.

[0107] According to one aspect of this disclosure, a method for storing a bit stream is provided, comprising performing the steps of any of the method embodiments described above to generate a bit stream and storing the bit stream.

[0108] According to one aspect of this disclosure, a computing device is provided, including a memory, at least one processor, and instructions stored in the memory, which, when executed by the at least one processor, cause the computing device to perform the steps of any of the method embodiments described above.

[0109] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0110] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0111] In the following text, combined with Figure 6 Illustrative examples describing such computing devices, non-transitory computer-readable storage media, and computer program products.

[0112] Figure 6 An example configuration of a computing device 600 that can be used to implement the methods described herein is shown.

[0113] The computing device 600 can be a variety of different types of devices. Examples of the computing device 600 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0114] The computing device 600 may include at least one processor 602, memory 604, multiple communication interfaces 606, display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612 capable of communicating with each other, such as via a system bus 614 or other suitable connection.

[0115] Processor 602 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 602 may be configured to acquire and execute computer-readable instructions stored in memory 604, mass storage device 612, or other computer-readable media, such as program code of operating system 616, program code of application program 618, program code of other program 620, etc.

[0116] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to perform the various functions described above. For example, memory 604 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 612 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to perform the operations and functions described in the examples herein.

[0117] Multiple programs can be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they can be loaded into memory 604 for execution. Examples of such application programs or program modules may include, for example, Figure 2 The computer program logic (e.g., computer program code or instructions) of the methods shown and / or other embodiments described herein.

[0118] Although Figure 6 The modules 616, 618, 620, and 622, or portions thereof, are illustrated as being stored in memory 604 of computing device 600; however, modules 616, 618, 620, and 622 may be implemented using any form of computer-readable medium accessible by computing device 600. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0119] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0120] One or more communication interfaces 606 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 606 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 606 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0121] In some examples, a display device 608, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0122] The techniques described herein can be supported by these various configurations of computing device 600, and are not limited to specific examples of the techniques described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on servers remote from computing device 600. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computing device 600 to other computing devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality may be implemented partly on computing device 600 and partly through a platform that abstracts the functionality of the cloud.

[0123] Although the present disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and schematic, not restrictive, and the present disclosure is not limited to the disclosed embodiments.

Claims

1. A method for real-time video encoding, characterized in that, The method includes: Receive image frames from the image frame sequence in real time from the rendering engine; Determine whether the current frame in the image frame sequence is a rendered frame or an interpolated frame; In response to determining that the current frame is a rendered frame, the current frame is encoded in a first encoding pipeline based on the encoding result of the previous rendered frame in the image frame sequence; and In response to determining that the current frame is an interpolated frame, motion information generated during the interpolation calculation of the current frame is obtained from the rendering engine, and the current frame is encoded in a second encoding pipeline different from the first encoding pipeline based on the encoding results of the two rendering frames immediately adjacent to the current frame in the image frame sequence and the motion information.

2. The method according to claim 1, characterized in that, The first encoding pipeline is configured to encode based on forward prediction, and the second encoding pipeline is configured to encode based on bidirectional prediction.

3. The method according to claim 1, characterized in that, The method further includes: For each rendered frame in the image frame sequence, the encoding result of the rendered frame is combined with the encoding result of the interpolation frame between the rendered frame and the next rendered frame in the image frame sequence to form a micro-image group for transmission in the same data packet.

4. The method according to claim 1, characterized in that, The first encoding pipeline and the second encoding pipeline belong to different video encoders.

5. A method for video decoding, characterized in that, The method includes: Receive the data packet generated by the method according to any one of claims 1 to 4; The data packets are decoded to obtain rendered frames and interpolated frames; and The rendered frames and the interpolated frames are reordered to obtain an image frame sequence.

6. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed by the processor of a computer, the processor performs the method according to any one of claims 1 to 5.

7. A method for storing a bit stream, characterized in that, Perform the method according to any one of claims 1 to 4 to generate a bit stream and store the bit stream.

8. A computing device, characterized in that, include: At least one processor; as well as A memory that stores instructions, which, when executed by the at least one processor, cause the computing device to perform the method according to any one of claims 1 to 5.

Images