Efficient quantization parameter prediction method for low latency video coding

Abstract

Disclosed are systems, devices, and methods for calculating a quantization parameter (QP) for encoding a video frame to meet a given bit budget. Control logic coupled to an encoder calculates a complexity indicator that represents a difficulty level for encoding a previous video frame. The complexity indicator is based at least in part on a first parameter associated with the previous video frame and corresponds to one or more of a variance, an intra-predictor, and an inter-to-intra ratio. The control logic then uses the complexity indicator to calculate a preferred QP for encoding a current video frame to meet a given bit budget. By using the preferred QP generated based on the complexity indicator, the encoder is able to make fewer QP adjustments during the frame. This helps to improve the visual quality of the resulting encoded video bitstream.

Description

Background Art

[0001] Description of Related Technology

[0002] Various applications perform encoding and decoding of image or video content. For example, video transcoding, desktop sharing, cloud gaming, and game viewing activities are some of the applications that include support for encoding and decoding of content. One of the parameters that determines how the content is encoded is the quantization parameter (QP). In various block-based video encoding schemes, such as those that comply with the H.264 standard, the QP adjusts how much spatial detail is retained during the encoding process. The QP selected for each video frame or each block of a frame is directly related to the size of the encoded video frame or the size of the encoded block. Selecting a lower QP value will retain more spatial detail while producing a larger encoded size. Selecting a higher QP value will result in the loss of more spatial detail while producing a smaller encoded size. Note that the term "quantization parameter" may also be more generally referred to as "quantization strength." Summary of the invention

[0003] A system comprising: an encoder configured to encode video frames of a video stream; control logic coupled to the encoder, wherein the control logic is configured to: calculate a complexity indicator representing a difficulty level for encoding a previous video frame; and calculate a quantization parameter (QP) value based on the complexity indicator and a preferred size of the encoded video frame; wherein the encoder is configured to: generate an encoded video frame representing a current video frame by setting the QP to the QP value calculated by the control logic; and transmit the encoded video frame to a decoder for display.

[0004] In some implementations, the complexity indicator is based at least in part on a first parameter associated with the previous video frame and corresponds to one or more of a variance, an intra predictor, and an inter-to-intra ratio.

[0005] In some implementations, the intra prediction factor indicates how effective intra prediction is for the previous video frame at a given QP.

[0006] In some implementations, the inter-to-intra ratio represents a ratio of inter-coded bits to intra-coded bits of the previous video frame at a given QP.

[0007] In some embodiments, the control logic is further configured to calculate the complexity indicator based on two or more previous video frames of the video stream.

[0008] In some implementations, the preferred size of the encoded video frame is calculated based on a preferred bit rate of the encoded video bitstream.

[0009] In some embodiments, the control logic is further configured to generate one or more parameters based on the current video frame for use in generating a next complexity indicator for encoding a next video frame.

[0010] A method comprising: calculating, by control logic, a complexity indicator based on a difficulty level for encoding a previous video frame; calculating, by the control logic, a quantization parameter (QP) value based on the complexity indicator and a preferred size of an encoded video frame; generating, by an encoder, an encoded video frame representing a current video frame by setting the QP to the QP value calculated by the control logic; and transmitting, by the encoder, the encoded video frame to a decoder for display.

[0011] In some implementations, the complexity indicator is based at least in part on a first parameter associated with the previous video frame and corresponds to one or more of a variance, an intra predictor, and an inter-to-intra ratio.

[0012] In some implementations, the intra prediction factor indicates how effective intra prediction is for the previous video frame at a given QP.

[0013] In some implementations, the inter-to-intra ratio represents a ratio of inter-coded bits to intra-coded bits of the previous video frame at a given QP.

[0014] In some embodiments, the method further comprises calculating the complexity indicator based on two or more previous video frames of the video stream.

[0015] In some implementations, the preferred size of the encoded video frame is calculated based on a preferred bit rate of the encoded video bitstream.

[0016] In some embodiments, the method further includes generating one or more parameters based on the current video frame for use in generating a next complexity indicator for encoding a next video frame.

[0017] A device comprising: a memory; an encoder coupled to the memory; and control logic coupled to the encoder, wherein the control logic is configured to: calculate a complexity indicator based on a difficulty level for encoding a previous video frame; and calculate a quantization parameter (QP) value based on the complexity indicator and based on a preferred size of the encoded video frame; wherein the encoder is configured to: generate an encoded video frame representing a current video frame by setting the QP to the QP value calculated by the control logic; and transmit the encoded video frame to a decoder for display.

[0018] In some implementations, the complexity indicator is based at least in part on a first parameter associated with the previous video frame and corresponds to one or more of a variance, an intra predictor, and an inter-to-intra ratio.

[0019] In some implementations, the intra prediction factor indicates how effective intra prediction is for the previous video frame at a given QP.

[0020] In some implementations, the inter-to-intra ratio represents a ratio of inter-coded bits to intra-coded bits of the previous video frame at a given QP.

[0021] In some embodiments, the control logic is configured to calculate the complexity indicator based on two or more previous video frames of the video stream.

[0022] In some implementations, the preferred size of the encoded video frame is calculated based on a preferred bit rate of the encoded video bitstream. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0024] Figure 1 is a block diagram of one implementation of a system for encoding and decoding content.

[0025] Figure 2 is a block diagram of one implementation of software components of a server for encoding frames of a video.

[0026] Figure 3 An equation for calculating a quantization parameter (QP) for encoding a video frame is illustrated according to one implementation.

[0027] Figure 4 Equations for calculating parameters associated with encoding of a frame are illustrated according to one implementation.

[0028] Figure 5 is a generalized flow chart illustrating one implementation of a method for determining a quantization parameter (QP) for use in encoding a video frame.

[0029] Figure 6 is a generalized flow chart illustrating another implementation of a method for calculating an estimated size of an encoded frame based on a complexity indicator.

[0030] Figure 7 is a generalized flow diagram illustrating one implementation of a method for determining a QP to use when encoding a video frame.

[0031] Figure 8 is a generalized flow chart illustrating one implementation of a method for calculating an estimated size of an encoded frame based on a complexity indicator. DETAILED DESCRIPTION

[0032] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, it will be appreciated by those of ordinary skill in the art that various implementations may be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid obfuscating the methods described herein. It will be appreciated that for ease and clarity of illustration, the elements shown in the figures may not be drawn to scale. For example, the size of some of the elements may be enlarged relative to other elements.

[0033] Disclosed herein are systems, devices, and methods for calculating a quantization parameter (QP) for encoding a video frame to meet a given bit budget. In one implementation, a system includes at least an encoder, control logic, and a memory coupled to the encoder and the control logic. The control logic calculates a complexity indicator based on at least a first parameter associated with a previous video frame and / or a first encoding pass of a current video frame. In general, the complexity indicator represents a difficulty level for encoding a given frame. In one implementation, the first parameter is a variance. In another implementation, the first parameter is an intra-frame prediction factor. In yet another implementation, the first parameter is an inter-frame to intra-frame ratio. In other implementations, the first parameter includes multiple of these parameters and / or one or more other parameters. In some implementations, the control logic calculates the complexity indicator based on at least the first parameter associated with two or more previous video frames. Then, the control logic uses the complexity indicator to calculate a preferred QP for encoding a current video frame to meet a given bit budget. By using the preferred QP generated based on the complexity indicator, the encoder is able to make fewer QP adjustments during the frame. This helps improve the visual quality of the resulting encoded video bitstream.

[0034] Reference now Figure 1 , a block diagram of one implementation of a system 100 for encoding and decoding content is shown. The system 100 includes a server 105, a network 110, a client 115, and a display 120. In other implementations, the system 100 includes a plurality of clients connected to the server 105 via the network 110, wherein the plurality of clients receive the same bitstream or different bitstreams generated by the server 105. The system 100 can also include more than one server 105 for generating a plurality of bitstreams for a plurality of clients.

[0035] In one implementation, the system 100 implements encoding and decoding of video content. In various implementations, the system 100 implements different applications, such as video game applications, cloud game applications, virtual desktop infrastructure applications, or screen sharing applications. In other implementations, the system 100 executes other types of applications. In one implementation, the server 105 renders video or image frames, encodes the rendered frames into a bitstream, and then transmits the encoded bitstream to the client 115 via the network 110. The client 115 decodes the encoded bitstream and generates a video or image frame to drive to the display 120 or display synthesizer.

[0036] The network 110 represents any type of network or combination of networks, including wireless connections, direct local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), intranets, the Internet, wired networks, packet switching networks, fiber optic networks, routers, storage area networks, or other types of networks. Examples of LANs include Ethernet networks, fiber distributed data interface (FDDI) networks, and token ring networks. In various implementations, the network 110 includes remote direct memory access (RDMA) hardware and / or software, transmission control protocol / Internet protocol (TCP / IP) hardware and / or software, routers, repeaters, switches, power grids, and / or other components.

[0037] The server 105 includes any combination of software and / or hardware for rendering video / image frames and encoding frames into bitstreams. In one implementation, the server 105 includes one or more software applications executed on one or more processors of one or more servers. The server 105 also includes network communication capabilities, one or more input / output devices and / or other components. The processor of the server 105 includes any number and type of processors (e.g., graphics processing unit (GPU), central processing unit (CPU), digital signal processor (DSP), field programmable gate array (FPGA), application-specific integrated circuit (ASIC)). The processor is coupled to one or more memory devices storing program instructions executable by the processor. Similarly, the client 115 includes any combination of software and / or hardware for decoding bitstreams and driving frames to the display 120. In one implementation, the client 115 includes one or more software applications executed on one or more processors of one or more computing devices. In various implementations, the client 115 is a computing device, a game console, a mobile device, a streaming media player or other types of devices.

[0038] Now go to Figure 2 , a block diagram of one implementation of software components of the server 200 for encoding frames of a video is shown. Note that in other implementations, the server 200 includes other components and / or is used in a manner similar to the embodiment of the present invention. Figure 2 The server 200 receives a new frame 205 of video and provides the new frame to the control logic 220 and the encoder 230. Each of the control logic 220 and the encoder 230 is implemented using any suitable combination of hardware and / or software. The control logic 220 generates a quantization parameter (QP) 225 used by the encoder 230 when encoding the new frame 205. In one implementation, the control logic 220 generates the QP 225 based on the parameters 210 for the previous frame and / or the parameters from the first pass encoding of the new frame 205. In various implementations, these parameters include one or more of variance, intra-frame prediction factors, inter-frame intra-frame ratios, and / or one or more other parameters. In one implementation, the control logic 220 generates new frame parameters 215 based on statistical information generated during the analysis and / or first pass encoding of the new frame 205. Control logic 220 stores new frame parameters 215 in memory 240, and new frame parameters 215 are used for a second pass encoding of new frame 205 and / or subsequent frames of the video stream. Memory 240 represents any number and type of memory or cache devices for storing data and / or instructions associated with an encoding process.

[0039] The encoder 230 receives the new frame 205 and encodes the new frame 205 using a QP value equal to the QP 225 generated by the control logic 220. In one implementation, when the encoder 230 begins encoding the new frame 205, the encoder sets the starting QP value equal to the QP 225 calculated by the control logic 220. The output of the encoder 230 is an encoded frame 235, which is transmitted to one or more clients (e.g., Figure 1 The control logic 220 performs a first pass encoding of the new frame 205 and the encoder 230 performs a second pass encoding of the new frame 205. In another implementation, the encoder 230 performs a first pass encoding and a second pass encoding of the new frame 205. In yet another implementation, the encoder 230 performs only a single pass encoding of the new frame 205.

[0040] In one implementation, the control logic 220 generates a complexity indicator that is based on the parameters 210 for the previous frame and / or the parameters from the first pass encoding of the new frame 205. The complexity indicator is then used to generate a starting QP value 225 for encoding the new frame 205 based on the bit rate that has been selected for the resulting encoded bitstream. By selecting an appropriate starting QP value 225, the encoder 230 will generally make fewer QP adjustments during encoding of the new frame 205. This helps improve the visual quality of the resulting encoded frame 235.

[0041] Reference now Figure 3 , an equation for calculating a quantization parameter (QP) for encoding a video frame is shown. The size of the compressed frame is calculated according to the formula shown in equation 305. The bit size of the compressed frame is equal to the complexity indicator divided by two QP divided by the value of α to the power of α. The complexity indicator is a value that indicates how difficult it is for the encoder to compress the frame. For the Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) video compression standards, the range of QP values ​​is 0 to 51. For the VP9 standard, the range of QP values ​​is 0 to 255. For a particular encoder, the α value is a constant value derived experimentally. Equation 310 is generated by solving equation 305 for QP.

[0042] In one implementation, to derive the complexity indicator for a given frame, the following parameters are determined. First, the variance of frame i, or variance(i), is determined. In addition, the complexity indicator representing the intra prediction at QP is determined. i The value of intra_pred_factor(i, QP i ). Please note that "intra_pred_factor(i, QP i)” is also referred to as “intra-frame prediction factor” in this article. In addition, QP is determined i The ratio of inter-bits to intra-bits for frame i under QP i ). Please note that "inter_intra_ratio(i, QP i )” is also referred to herein as “inter-to-intra ratio”. For these parameters, in one implementation, “i” is the number of the previously encoded frame. In another implementation, “i” is the first pass number of the frame currently being encoded. Similarly, in one implementation, “QP i " is the QP value for the previously encoded frame, and in another implementation, "QP i " is the QP value for the first pass of the frame currently being encoded. In a single-pass encoder, the three parameters (variance(i), intra_pred_factor(i, QP i )、inter_intra_ratio(i,QP i )) will be accumulated from previously encoded frames. In a secondary encoder, these three parameters will be accumulated in the first pass.

[0043] In one implementation, one of the components of the "complex" value is a measure of the information contained in the frame. In one implementation, the value used to measure the information contained in the frame is activity. In one implementation, activity is calculated by adding the sum of the absolute differences between adjacent pixels. In another implementation, variance is used to measure the information contained in the frame. In this implementation, the variance of a frame is determined by the following equation: variance(i) = ∑ b block_variance(b). In this equation, "b" is the block number in the frame, and the size of the block is 16x16 pixels, 32x32 pixels, 64x64 pixels, or other sizes. Also, in this equation, block_variance(b) is the variance of the pixel values ​​in block "b".

[0044] Now go to Figure 4 , shows the equations for calculating the parameters associated with the encoding of a frame. intra_pred_factor(i, QP i ) value indicates how effective intra prediction is for a frame. As used herein, "intra prediction" refers to predicting pixel values ​​within a given frame based on other pixel values ​​within the same frame. When encoding a frame, the encoder supplies the number of bits required to encode the intra blocks (at QP iThis value will be referred to as intraBits(b), and intraBits(b) is one of the statistics used in the rate-distortion optimization (RDO) process of the encoder. In one implementation, intra_pred_factor(i, QP i ).

[0045] For inter-frames, inter_intra_ratio(i, QP i ) value is the ratio of required inter-frame bits to required intra-frame bits (at QP i As used herein, the term "inter bits" refers to the number of bits required to encode a frame using only inter-frame coding. The term "inter-frame coding" is defined as encoding a frame using information from one or more other frames before and / or after the frame and no information from other pixels within the frame. As used herein, the term "intra bits" refers to the number of bits required to encode a frame using only intra-frame coding (i.e., intra-frame prediction). The encoder provides the interBits(b) value in the same manner as the intraBits(b) value. In one implementation, inter_intra_ratio(i, QP i ) value is derived from equation 410.

[0046] Once a given frame has been encoded, the following parameters will be available for frame i: variance(i), intra_pred_factor(i, QP i )、inter_intra_ratio(i,QP i ) and size(i, QP i ).size(i,QP i ) parameter refers to the size of the encoded frame i in bits. In one implementation, the actual frame complexity value ("actual_complexity") is derived using equation 415. In one implementation, the estimated complexity ("estimated_complexity") value of an intra frame is derived using equation 420: estimated_complexity = variance(i)*intra_pred_factor(i, QP i In one implementation, the estimated complexity (“estimated_complexity”) value of the inter-frame is derived using equation 425: estimated_complexity=variance(i)*intra_pred_factor(i,QP i )*inter_intra_ratio(i,QP i ).

[0047] In one implementation, if the preferred size (or SIZE) of the compressed frame is known, then, if size(i, QP i ) < SIZE, the following formula shown in Equation 430 is used to calculate the weighting factor: weight = size(i, QP i ) / SIZE. Otherwise, if size(i, QP i ) ≥ SIZE, the following formula shown in Equation 435 is used to calculate the weighting factor: weight = SIZE / size(i, QP i ). In one implementation, if the preferred QP (or QP preferred ) is known, then, if QP i < QP preferred , the following formula shown in Equation 440 is used to calculate the weighting factor: weight = QP i / QP preferred . In one implementation, if QP i ≥ QP preferred , the following formula shown in Equation 445 is used to calculate the weighting factor: weight = QP preferred / QP i . In one implementation, the following formula shown in Equation 450 is used to calculate the complexity indicator: complex = (weight * actual_complexity) + ((1 - weight) * estimated_complexity). Once the complexity indicator is calculated, the SIZE or QP value is derived using Equation 305 or 310, respectively.

[0048] Now referring to Figure 5 , an implementation of a method 500 for determining a quantization parameter (QP) to be used when encoding a video frame is shown. For purposes of discussion, the steps in this implementation and Figures 6 to 8 those steps are shown in sequential order. However, note that in various implementations of the described method, one or more of the described elements are performed simultaneously, performed in a different order than shown, or omitted entirely. Other additional elements are also performed as needed. Any of the various systems or devices described herein are configured to implement method 500.

[0049] The control logic connected to the encoder calculates the complexity indicator based on at least a first parameter associated with a previous video frame. As previously described, the complexity indicator represents the difficulty level of encoding the previous video frame (box 505). In one implementation, the first parameter is a variance. In another implementation, the first parameter is an intra-frame prediction factor. In yet another implementation, the first parameter is an inter-frame to intra-frame ratio. In other implementations, the first parameter includes a plurality of these parameters and / or one or more other parameters. In some implementations, the control logic calculates the complexity indicator based on at least the first parameter associated with two or more previous video frames.

[0050] Next, the control logic calculates a quantization parameter (QP) value based on the complexity indicator and based on the preferred size of the encoded video frame (block 510). In one implementation, using ( Figure 3 ) equation 310 to calculate the QP value. In one implementation, the preferred size of the encoded video frame is determined based on the preferred bit rate for transmitting the encoded video stream. For example, in one implementation, the preferred bit rate is specified in bits per second (e.g., 3 megabits per second (Mbps)), and the frame rate of the video sequence is specified in frames per second (fps) (e.g., 60fps, 24fps). In this implementation, the control logic divides the preferred bit rate by the frame rate to calculate the preferred size of the encoded video frame. The encoder then sets the QP for encoding the current video frame to the QP value calculated by the control logic (block 515). Next, the encoder encodes the current video frame using the calculated QP value and transmits the encoded current video frame to the decoder (block 520). After block 520, the method 500 ends.

[0051] Now go to Figure 6 , shows one implementation of a method 600 for calculating an estimated size of an encoded frame based on a complexity indicator. Control logic connected to or integrated within an encoder calculates a complexity indicator based on at least a first parameter associated with a previous video frame, wherein the complexity indicator represents a difficulty level for encoding the previous video frame (block 605). Next, the control logic calculates an estimated size of the encoded video frame based on the complexity indicator and based on a given QP for encoding the video frame (block 610). After block 610, the method 600 ends.

[0052] Reference now Figure 7, shows one implementation of a method 700 for determining a QP to be used when encoding a video frame. Control logic associated with an encoder calculates a complexity indicator based on at least a first parameter associated with a first pass encoding of a given video frame, wherein the complexity indicator represents a difficulty level for the first pass encoding of the given video frame (block 705). Next, the control logic calculates a quantization parameter (QP) value based on the complexity indicator and based on a preferred size for a final encoded given video frame (block 710). The final encoded given video frame refers to an output of a second pass encoding of the given video frame. The encoder then sets the QP for the second pass encoding of the given video frame to the QP value calculated by the control logic (block 715). After block 715, the method 700 ends.

[0053] Now go to Figure 8 , shows one implementation of a method 800 for calculating an estimated size of an encoded frame based on a complexity indicator. Control logic coupled to an encoder calculates a complexity indicator based on at least a first parameter associated with a first pass encoding of a given video frame, wherein the complexity indicator represents a difficulty level of the first pass encoding of the given video frame (block 805). Next, the control logic calculates an estimated size of a final encoded given video frame based on the complexity indicator and based on a given QP for the first pass encoding of the given video frame (block 810). After block 810, the method 800 ends.

[0054] In various implementations, the method and / or mechanism described herein is implemented using program instructions of a software application. For example, program instructions that can be executed by a general or special processor are contemplated. In various implementations, such program instructions can be represented by a high-level programming language. In other implementations, program instructions can be compiled from a high-level programming language into a binary form, an intermediate form, or other form. Alternatively, program instructions describing the behavior or design of hardware can be written. Such program instructions can be represented by a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog can be used. In various implementations, program instructions are stored on any of a variety of non-temporary computer-readable storage media. During use, a computing system can access a storage medium to provide program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

[0055] It should be emphasized that the above implementations are only non-limiting examples of implementations. Once the above disclosure is fully understood, many changes and modifications will become apparent to those skilled in the art. The attached claims are intended to be interpreted as covering all such changes and modifications.

Claims

1. A system, comprising: an encoder configured to encode video frames of a video stream; a control logic coupled to the encoder, wherein the control logic is configured to: computing a complexity indicator representing a difficulty level for encoding a previous video frame based at least in part on: quantization parameter QP of the previous video frame; and the calculated variance of the previous video frame, calculating the complexity indicator; and Calculating a quantization parameter QP value based on the complexity indicator and a preferred size of the encoded video frame; The encoder is configured to: generating an encoded video frame representing a current video frame by setting the QP to the QP value calculated by the control logic; and The encoded video frames are transmitted to a decoder for display.

2. The system of claim 1, wherein the complexity indicator is further based on one of a calculated intra predictor of the previous video frame and a calculated inter-to-intra ratio of the previous video frame.

3. The system of claim 2, wherein the intra prediction factor is calculated based on intra bits used to encode the previous video frame.

4. The system of claim 3, wherein the inter-to-intra ratio is calculated based on a ratio of inter bits used to encode the previous video frame to intra bits used to encode the previous video frame, wherein the inter bits correspond to bits required to encode a frame using only inter coding.

5. The system of claim 1, wherein the variance is calculated based on a sum of variances of blocks in the previous video frames.

6. The system of claim 1, wherein the control logic is further configured to generate one or more parameters based on the current video frame for use in generating a next complexity indicator for encoding a next video frame.

7. A method comprising: The complexity indicator is calculated by the control logic based on a difficulty level for encoding a previous video frame, based at least in part on: Calculating the complexity indicator based on a quantization parameter QP of the previous video frame and a calculated variance of the previous video frame; Calculating, by the control logic, a quantization parameter (QP) value based on the complexity indicator and a preferred size of an encoded video frame; generating, by the encoder, an encoded video frame representing a current video frame by setting the QP to the QP value calculated by the control logic; as well as The encoded video frames are transmitted by the encoder to a decoder for display.

8. The method of claim 7, wherein the complexity indicator is further based on one of a calculated intra predictor of the previous video frame and a calculated inter-to-intra ratio of the previous video frame.

9. The method of claim 8, wherein the intra predictor is calculated based on intra bits used to encode the previous video frame.

10. The method of claim 9, wherein the calculation is based on a ratio of inter bits used to encode the previous video frame to intra bits used to encode the previous video frame, wherein the inter bits correspond to bits required to encode a frame using only inter coding.

11. The method of claim 7, further comprising calculating the variance based on a sum of variances of blocks in the previous video frames.

12. The method of claim 7, further comprising generating one or more parameters based on the current video frame for use in generating a next complexity indicator for encoding a next video frame.

13. A device, comprising: Memory; as well as A system according to any one of claims 1 to 6.

Images