Measuring bits in a video encoder to achieve a target data rate and video quality level

Abstract

To stabilize the bit rate of a group of pictures (GOP), at least one video buffer can be maintained using a rate buffer bit controller feedback loop and a proportional / integral / derivative (PID) bit controller feedback loop (700).

Description

[Technical Field]

[0001] This application relates generally to measuring bits in a video encoder to achieve a target data rate and video quality level. [Background technology]

[0002] Virtual cloud networks (VCNs) can be used for network video distribution. However, as understood herein, problems can arise because the network bitrate even from one group of pictures (GOP) to another is unstable, and the input buffer to the encoder can become overfilled or underfilled. To date, there are no adequate solutions to the above computer-related technical problems. Summary of the Invention

[0003] A GOP may be a short sequence of video frames within a video stream (e.g., a stream of a single video content such as a movie or internet video). As understood herein, VCN firmware may be implemented to steadily measure and maintain a desired bit rate per GOP for quality. Thus, a device executing consistent with the present principles may optimally keep the output buffer at a desired level of bits ready for output for display without over- or under-filling.

[0004] To that end, in one aspect, an apparatus includes at least one processor configured to maintain at least one video buffer according to a rate buffer bit controller feedback loop and a proportional integral derivative (PID) bit controller feedback loop. The PID bit controller feedback loop includes a first branch, a second branch, and a third branch in parallel with one another, each receiving a feedback error. The first branch is configured to modify a target bit rate based at least in part on the feedback error over a first period. The second branch is configured to modify the target bit rate based at least in part on the feedback error over a second period longer than the first period. The third branch is configured to reduce overshoot.

[0005] In some example implementations, the at least one processor may also be configured to use a first predefined quantization parameter (QP) for a first I-frame of a group of pictures (GOP), which may be the first I-frame of the GOP. In these exemplary implementations, the at least one processor may also be configured to use a second QP for a second I-frame of the GOP, which may be determined by the at least one processor from a third QP of a previous I-frame in the GOP and a fourth QP of a last non-I-frame of the GOP that preceded the second I-frame. The third QP may or may not be different from the first QP. The previous I-frame may be the last I-frame in the GOP that preceded the second I-frame, and the second QP may be determined from an average of the third QP and the fourth QP. Further, optionally, the second I-frame may be a frame from the same scene as the first I-frame, which may be the first scene, and the at least one processor may be configured to use a fifth QP for the third I-frame of the GOP. The third I-frame may be a frame for a second scene different from the first scene, and the fifth QP may be determined by the at least one processor from an average of two or more QPs of previous I-frames from the GOP. The two or more QPs of the previous I-frames may be associated with the first scene.

[0006] Furthermore, in some examples, the processor may be configured to determine a QP for non-I frames based on execution of a rate buffer bit controller feedback loop and a PID bit controller feedback loop.

[0007] Furthermore, in some examples, the first branch may be configured to adjust a first constant K to modify the target bit rate based at least in part on the feedback error over the first time period. P and a second branch may use a second constant K to modify the target bit rate based at least in part on the feedback error over a second time period. I The third branch uses a third constant, K, to reduce overshoot. D Thus, for example, the first constant, the second constant, and the third constant may be non-negative numbers determined using a machine learning based model, and the first branch may be a function K P (PIDE(n)) may be applied, and the second branch is the function

number

[0008] In certain exemplary embodiments, the apparatus may include an encoder device, and the at least one video buffer may be maintained by at least one processor in the encoder device. Additionally or alternatively, the apparatus may include a decoder device, and the at least one video buffer may be maintained by at least one processor in the decoder device.

[0009] In another aspect, a method implements a rate buffer bit controller feedback loop, implements a proportional / integral / derivative (PID) bit controller feedback loop, and maintains at least one video buffer according to the rate buffer bit controller feedback loop and the proportional / integral / derivative (PID) bit controller feedback loop.

[0010] In a particular example, the method may process a first feedback error over a first time period via a first branch of a PID bit controller feedback loop to modify the target bit rate, a second feedback error over a second, longer time period via a second branch of the PID bit controller feedback loop to modify the target bit rate, and a third feedback error via a third branch of the PID bit controller feedback loop to reduce overshoot. Thus, in one exemplary implementation, the first branch adjusts a first constant K to modify the target bit rate based at least in part on a first feedback error over a first time period. P and a function (PIDE(n)), wherein the second branch uses a second constant K to modify the target bit rate based at least in part on a second feedback error over a second time period. I and functions

number

[0011] Also, in some exemplary implementations, the method may use a predefined first quantization parameter (QP) for a first I-frame of a group of pictures (GOP), which may be the first I-frame of the GOP. In these exemplary implementations, the method may also use a second QP for a second I-frame of the GOP, which may be determined from the average of a third QP of a previous I-frame in the GOP and a fourth QP of the last non-I-frame of the GOP that preceded the second I-frame.

[0012] In yet another aspect, a device includes at least one computer storage device, the computer storage device including instructions executable by at least one processor to use a proportional / integral / derivative (PID) bit controller feedback loop to maintain at least one video buffer rather than a transient signal. The PID bit controller feedback loop includes a first branch, a second branch, and a third branch in parallel with one another, each receiving a feedback error. The first branch is configured to modify a target bit rate based at least in part on the feedback error over a first period. The second branch is configured to modify the target bit rate based at least in part on the feedback error over a second period longer than the first period. The third branch is configured to reduce overshoot.

[0013] In some example embodiments, the instructions may also be executable to use a rate buffer bit controller feedback loop and a PID bit controller feedback loop to maintain at least one video buffer.

[0014] Also, in some exemplary embodiments, the first branch adjusts a first constant K to modify the target bit rate based at least in part on the feedback error over the first time period. P and a function (PIDE(n)), and the second branch may use a second constant K to modify the target bit rate based at least in part on the feedback error over a second time period. I and functions

number

[0015] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram of an exemplary system in accordance with present principles; [Figure 2] 1 is a block diagram of an exemplary encoder device consistent with the present principles; [Figure 3] 1 is a block diagram of an exemplary decoder device consistent with the present principles; [Figure 4] FIG. 1 is a schematic diagram of an exemplary software architecture consistent with the present principles. [Figure 5] 1 illustrates, in exemplary flow chart form, exemplary logic executable by an encoder device consistent with the present principles. [Figure 6] 1 illustrates, in exemplary flow chart form, exemplary logic executable by a decoder device consistent with the present principles. [Figure 7] 1 illustrates an exemplary proportional / integral / derivative (PID) bit controller feedback loop consistent with the present principles. [Figure 8]1 illustrates exemplary logic for an exemplary TFB(n) algorithm consistent with the present principles in exemplary flow chart form. [Figure 9] 1 illustrates exemplary overall logic for rate control consistent with the present principles. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present disclosure relates generally to computer ecosystems, including, but not limited to, aspects of consumer electronics (CE) device networks, such as computer gaming networks. The systems herein may include server and client components that may be connected over a network such that data may be exchanged between the client and server components. The client component may include one or more computing devices, including game consoles such as a Sony PlayStation®, or game consoles made by Microsoft or Nintendo, or other manufacturers, extended reality (XR) headsets such as virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, internet-enabled televisions), portable computers such as laptop computers and tablet computers, and smartphones and other mobile devices, including additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may use, as examples, the Linux operating system, an operating system manufactured by Microsoft, or a Unix operating system, or an operating system manufactured by Apple, Inc., or Google, or a Berkeley Software Distribution or Berkeley Standard Distribution (BSD) OS (including derivatives of BSD). These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs capable of accessing websites hosted by the Internet servers described below. An operating environment according to present principles may also be used to run one or more computer game programs.

[0018] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected via a local intranet or virtual private network. The server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, or the like.

[0019] Information may be exchanged between the client and the server over a network. To this end, and for security, the server and / or client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website or gamer network, to network members.

[0020] The processor may be a single-chip processor or a multi-chip processor capable of performing logic through various lines, such as address lines, data lines, and control lines, as well as registers and shift registers. A processor, including a digital signal processor (DSP), may be an embodiment of a circuit.

[0021] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or depicted in the figures may be combined, substituted, or excluded from other embodiments.

[0022] "A system having at least one of A, B, and C" (and similarly, "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes a system having only A, a system having only B, a system having only C, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having both A, B, and C.

[0023] 1, an exemplary system 10 is shown. System 10 may include one or more of the exemplary devices in accordance with the present principles referenced above and further described below. A first of the exemplary devices included in system 10 is a consumer electronics (CE) device such as an audio-video device (AVD) 12, such as, but not limited to, a theater display system, which may be projector-based, or an Internet-enabled television with a TV tuner (or, equivalently, a set-top box that controls a TV). Alternatively, AVD12 may also be a computerized internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or headset (such as smart glasses or a VR headset), another computerized wearable device, a computerized internet-enabled music player, computerized internet-enabled headphones, a computerized internet-enabled implantable device (such as an implantable skin device), or the like. In any event, it will be understood that AVD12 is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0024] Accordingly, to implement such principles, AVD 12 may be established by some or all of the components shown. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by high-definition or ultra-high-definition "4K" or higher flat screens. Touch-enabled display(s) 14 may include, for example, a capacitive or resistive touch-sensing layer with a grid of electrodes for touch sensing consistent with the present principles.

[0025] AVD 12 may also include one or more speakers 16 for outputting audio in accordance with present principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to AVD 12 for controlling AVD 12. Example AVD 12 may also include one or more network interfaces 20, under the control of one or more processors 24, for communicating over at least one network 22, such as the Internet, a WAN, a LAN, or the like. Thus, interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It will be understood that processor 24 controls AVD 12 to implement the present principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images on display 14 and receiving input from display 14. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or a wireless telephony transceiver or other suitable interface, such as the Wi-Fi transceiver described above.

[0026] In addition to the foregoing, AVD 12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI®) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 for presenting audio from AVD 12 to a user via headphones. For example, input port 26 may be wired or wirelessly connected to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. If implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0027] The AVD 12 may further include one or more computer memory / computer-readable storage media 28, such as disk-based or solid-state storage, that are not transitory signals and may in some cases be embodied within the AVD's chassis as a standalone device, or as a personal video recording device (PVR) or video disc player either inside or outside the AVD's chassis for playing AV programs, or as a removable storage medium or a server as described below. Additionally, in some embodiments, AVD 12 may include a position or location receiver such as, but not limited to, a cellular telephone receiver, a GPS receiver, and / or an altimeter 30 configured to receive geographic location information from a satellite or cellular base station and provide that information to processor 24 and / or configured in conjunction with processor 24 to determine the altitude at which AVD 12 is located.

[0028] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be a thermal imaging camera, a digital camera such as a webcam, an IR sensor, an event-based sensor, and / or a camera integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or video in accordance with the present principles. AVD 12 may also include a Bluetooth transceiver 34 and other NFC elements 36 for communicating with other devices using Bluetooth and / or near field communication (NFC) technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0029] Furthermore, AVD 12 may include one or more auxiliary sensors 38 that provide input to processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of touch-enabled display 14 itself, and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Examples of other sensors include pressure sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, and gesture sensors (e.g., sensors for sensing gesture commands). Thus, sensors 38 may be implemented by inertial measurement units (IMUs), which typically include one or more motion sensors, such as individual accelerometers, gyroscopes, and magnetometers, and / or combinations of accelerometers, gyroscopes, and magnetometers, to determine the position and orientation of AVD 12 in three dimensions, or by event-based sensors, such as event detection sensors (EDSs). An EDS consistent with this disclosure provides an output indicative of a change in light intensity sensed by at least one pixel of the light-sensing array. For example, if the light sensed by the pixel is decreasing, the output of the EDS may be −1. If increasing, the output of the EDS may be +1. No change in light intensity below a certain threshold may be indicated by an output binary signal of 0.

[0030] AVD 12 may also include an over-the-air TV broadcast port 40 for receiving OTA TV broadcasts that provide input to processor 24. Note that in addition to the foregoing, AVD 12 may also include an infrared (IR) transmitter and / or IR receiver and / or IR transceiver 42, such as an Infrared Data Association (IRDA) device. AVD 12 may be provided with a battery (not shown) for powering it, which may be a kinetic energy harvester that can convert kinetic energy into electrical power to charge the battery and / or power AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more haptic / vibration generators 47 may be provided to generate haptic signals that can be sensed by a person holding or interacting with the device. Thus, the haptic generator 47 may vibrate all or part of the AVD 12 using an electric motor connected to an off-center and / or unbalanced weight via the motor's rotating shaft, the shaft rotating under the control of the motor (which may be controlled by a processor such as processor 24) to create simulations of vibrations of various frequencies and / or amplitudes, and forces in various directions.

[0031] A light source such as a projector, such as an infrared (IR) projector, may also be included.

[0032] In addition to AVD 12, system 10 may include one or more other CE device types. In one example, first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server, as described below, while second CE device 50 may include similar components to first CE device 48. In the illustrated example, second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. HMDs may include heads-up transparent or heads-up opaque displays that present AR / MR or VR content (more generally, extended reality (XR) content), respectively. HMDs may be configured as glasses-type displays or as large VR-type displays sold by computer gaming device manufacturers.

[0033] In the illustrated example, only two CE devices are shown, and it will be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the figures below may incorporate some or all of the components shown in the AVD 12 example.

[0034] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with the other illustrated devices over the network 22 and, indeed, may facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephony transceiver.

[0035] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" that includes and is capable of performing "cloud" functionality, allowing devices of system 10 to access a "cloud" environment via server 52, for example, in embodiments for network gaming applications, or server 52 may be implemented on one or more game consoles or other computers in the same room or nearby as the other devices shown.

[0036] The components shown in the following figures may include some or all of the components shown herein. Any user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0037] The present principles may use a variety of machine learning models, including deep learning models. Machine learning models consistent with the present principles may use a variety of algorithms trained using methods including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms may be implemented by computer circuitry, including one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and a type of RNN known as a long short-term memory (LSTM) network. Support vector machines (SVMs) and Bayesian networks may also be considered examples of machine learning models. In addition to the types of networks listed above, the models herein may also be implemented by classifiers.

[0038] As understood herein, performing machine learning may include accessing and training a model on training data to enable the model to process additional data and make inferences. As a result, an artificial neural network / artificial intelligence model trained through machine learning may include a weighted input layer, an output layer, and multiple hidden layers in between, configured to make inferences about a suitable output.

[0039] Referring now to Figure 2, at least one exemplary transmitter / encoder device 200 is shown. Device 200 may be established by the devices of Figure 1, such as one or more servers that maintain and manage a VCN that provides GOPs of video. The video may be established by computer game video or other interactive video, or by video of other audio / video content, such as movies, internet-based tube video, video embedded in websites, video streamed to set-top boxes via cable and satellite connections, video provided via dedicated software applications (apps), etc.

[0040] 2 illustrates a video source 202 of video, which may be established by persistent storage, such as a hard disk drive and / or solid-state drive, on device 200 or a cloud server or other source. Thus, a processor (not shown) within device 200 may access the video from video source 202, pass the video through encoder 204 to encode GOPs of the video, and provide the encoded GOPs to a network interface 206 of device 200 for transmission to one or more receiver / decoder devices. Network interface 206 may be a computer network interface, a wireless telephony interface, a television broadcast interface, a satellite or terrestrial broadcast interface, or the like.

[0041] Referring now to Figure 3, at least one exemplary receiver / decoder device 300 is shown. Device 300 may receive the video of Figure 2 via GOPs from transmitting device 200. Device 300 may be established by a device of Figure 1, such as, for example, a personal computer such as a desktop computer, a laptop computer, a smartphone, a headset such as smart glasses or an augmented reality or virtual reality headset. Additionally or alternatively, device 300 may be established by a computer game console, a television, a set-top box, or the like.

[0042] 3, device 300 may include a network interface 302 that can receive video / GOPs from device 200 via a wired LAN or Wi-Fi Internet connection, a cable or satellite connection, a 5G or other wireless cellular Internet connection, etc. Thus, in various examples, network interface 302 may be a computer network interface, a wireless telephony interface, a television broadcast interface, or a satellite or terrestrial broadcast interface, etc.

[0043] When interface 302 receives a GOP from device 200, interface 302 may provide the GOP to decoder 304 for decoding. Decoder 304 may then provide the decoded GOP to a central processing unit (CPU) or other processor of device 300 and / or to a display driver controlling a display connected to device 300, which may place the GOP in a display buffer maintained by device 300.

[0044] Continuing with the detailed description, referring to Figure 4, a schematic diagram of an exemplary software architecture consistent with the present principles is shown. Encoder device 400 (which may be established by a VCN device such as device 200) has access to video 402, allowing software module 404 to calculate rate distortion and then allocate and update target frame bits. Module 404 may also maintain a rate buffer and calculate quantization parameters / values, as described further below. Module 404 may then provide its calculation results / output to software module 406, which may then perform entropy coding and packetization. Entropy coding may include Huffman coding and other techniques.

[0045] The encoded packets may then be transmitted from module 406 over one or more network channels 408 (e.g., over the Internet or another network) to decoder device 410. Decoder device 410 may be established with a VCN device, such as device 300. Software module 412 may then perform de-packetization and entropy decoding on the received packets before providing its output to software module 414. Module 414 may then maintain its rate buffer, calculate quantization parameters / values, decode each frame from the decoded packets, and place the frames in a display buffer for the display that is used to ultimately present the frames / video 402.

[0046] The example logic of Figures 5 and 6 will be further explained by way of example. Figure 5 shows the logic of an encoder device, and Figure 6 shows the logic of a decoder device. Referring first to Figure 5, in block 500, the encoder device may calculate rate distortion and then proceed to block 502. In block 502, the encoder device may allocate and update target frame bits. The logic may then proceed to block 504, where the encoder device may maintain its rate buffer and then calculate one or more quantization parameters / values ​​consistent with the present principles at block 506. The logic may then proceed to block 508, where the encoder device may perform entropy coding and packetization and then output the encoded packets at block 510 over a channel, such as channel(s) 408 described above.

[0047] 6, at block 600, the decoder device may receive the encoded packets received over the channel(s) output at block 510. The logic may then proceed to block 602, where the decoder device may perform de-packetization and entropy decoding. The logic may then proceed to block 604, where the decoder device maintains its rate buffer, and then proceed to block 606, where the decoder device may calculate one or more quantization parameters / values ​​consistent with the present principles. The logic may then proceed to block 608, where the decoder device may decode one or more frames and place the decoded frames in a display buffer for output via a display controlled by the decoder device, at block 610. The display may be, for example, an LED, LCD, or OLED display.

[0048] 6, it should be appreciated that a rate control algorithm consistent with the present principles may have two feedback loops. Specifically, the algorithm may use a PID bit controller and a rate buffer bit controller. Bit allocation for I-frames and non-I-frames (such as P-frames and B-frames) may be performed as follows:

[0049] For I-frames, a predetermined / predefined QP may be used for the initial / first frame of a GOP (a separate I-frame) and may be set by a designer, programmer, etc. Thus, the QP for this frame may be fixed (e.g., the frame does not undergo any processing to obtain a QP value). The remaining I-frames and non-I-frames of the GOP may undergo the loop process described herein to obtain QP values. Thus, for periodic I-frames of the remaining frames in the same GOP that are for the same scene in the GOP, the QP may be calculated as the average of the QP of the last frame and the QP of the last I-frame. For non-periodic I-frames of the same GOP but that establish a scene change / different scene, the QP may be calculated as the average QP from previous I-frames of the same GOP. And then for non-I-frames, the QP may be derived from a PID-based rate control algorithm.

[0050] Thus, according to Figure 7, the algorithm can maintain at least one video buffer according to a rate buffer bit controller feedback loop (not shown in Figure 7) and a proportional / integral / derivative (PID) bit controller feedback loop 700. The PID bit controller feedback loop 700 can include first, second, and third branches 702-706 in parallel with each other, each receiving a feedback error 708. The feedback error can be PIDE(n), as shown in Figure 7.

[0051] The first branch 702 may be configured to modify the target bit rate based at least in part on the feedback error over a first shorter period of time, the second branch 704 may be configured to modify the target bit rate based at least in part on the feedback error over a second longer period of time, and the third branch 706 may be configured to reduce overshoot.

[0052] Thus, as shown in FIG. 7, a first branch 702 calculates a first constant K to modify the target bit rate based at least in part on the feedback error over a first period of time. P and a second branch 704 may use a second constant K to modify the target bit rate based at least in part on the feedback error over a second period of time. I A third branch 706 can be used to reduce overshoot by adding a third constant K D The first constant, the second constant, and the third constant may be positive numbers / control parameters determined using a machine learning based model that learns and / or empirically determines the optimal respective constants.

[0053] As also shown in FIG. 7, PIDE may be the PID error, and therefore the first branch 702 is a function K P (PIDE(n)), and the second branch 704 applies the function

number

[0054] Thus, the PID-based bit controller shown in FIG. 7 may use a proportional / integral / derivative methodology such that the overall logic executed by the associated device may use the PID bit controller algorithm of FIG. 7 in conjunction with a rate control algorithm (e.g., a rate buffer bit controller feedback loop as further described below) in controlling the rate buffer, allocating channel bits, and determining quantization values ​​consistent with the present principles.

[0055] Therefore, K p can be a powerful control ability to correct the target in proportion to the error in the short term. i can reduce errors in the long run. d may reduce the overshoot. The following equation may then be used as part of a PID controller: PIDCB(n) = PID controller bits for the nth frame PIDCB(n)=PIDK P (n)+PIDK I (n)+PIDK D (n) PIDK P (n)=K P PIDE(n)

number

[0056] As previously mentioned, a rate control algorithm for maintaining a video buffer consistent with the present principles may have two feedback loops / controllers: a PID bit controller and a rate buffer controller. A target bit allocation for any frame may be calculated based on these two feedback controllers. Since stability of these two controllers is desirable, the present principles may be used to prevent instability under any circumstances. Therefore, as recognized herein, a constant factor K p , K. i , and K d may control the stability of the PID controller. In this way, the rate buffer overflow, underflow mechanism(s) may control the stability of the rate buffer controller.

[0057] A predefined QP may be used for every first I-frame in a GOP, and the remaining I-frames in the same GOP, except for scene change I-frames, may use the average of the QP of the previous frame and the QP of the last reference frame. I-frames due to scene changes can use the average of the QPs of past I-frames (e.g., from the same GOP).

[0058] The QP for P-frames and other non-I-frames may be derived from the PID and rate buffer control target bit allocation, as further described with reference to FIG. P , K. I , and K D is a constant that controls the stability of the PID controller and can be derived from an ML-based model as described above. The stability of the rate buffer is controlled by the buffer occupancy or buffer fullness, which is also indirectly controlled by the PID controller.

[0059] The overall process must remain stable to meet channel rate requirements, and therefore video quality must be maintained. Thus, this process can be used to maintain a relatively constant and efficient channel / bit rate and video quality for a given video, across multiple GOPs, and per GOP, under all circumstances.

[0060] Thus, in an exemplary implementation, a first predefined QP may be used for the first I-frame of a GOP, which may be the first I-frame of the GOP, and a second QP may be used for the second I-frame of the GOP, which may be determined from a third QP of the previous I-frame in the GOP and a fourth QP of the last non-I-frame of the GOP that preceded the second I-frame (preceded GOP). The third QP may or may not be different from the first QP. The previous I-frame may be the last I-frame in the GOP that preceded the second I-frame, and the second QP may be determined from an average of the third QP and the fourth QP. Further, the second I-frame may be a frame from the same scene as the first I-frame, which may be the first scene, and the at least one processor may be configured to use a fifth QP for the third I-frame of the GOP. The third I-frame may be a frame for a second scene different from the first scene, and the fifth QP may be determined by the at least one processor from an average of the QPs of two or more preceding I-frames from the GOP. The two or more QPs of the preceding I-frames may be associated with the first scene. QPs may also be determined for non-I-frames based on execution of a rate buffer bit controller feedback loop and a PID bit controller feedback loop. Therefore, please note the following: Channel rate in bits (bps) = CR Frame rate (fps) = FR Frame Channel Bit Average = FCB ave =(C R / F R ) PID controller bit = PIDCB(n) Target Frame Bits = TFB(n) = (TargetBitAllocation(n) + PIDCB(n))

number

[0061] With the above in mind, reference is now made to FIG. 8, which illustrates, in flow chart form, example logic for an example TFB(n) algorithm consistent with the present principles. Beginning at block 800, a device may determine whether TFB(n) is greater than zero. A negative determination at block 800 can cause the logic to proceed to block 802, where the TFB(n)=RD model updates bits (the rate-distortion and rate buffer control algorithms calculate QP and update bits).

[0062] However, in response to a positive determination at block 800, the logic may instead proceed to block 804. At block 804, the device determines whether TFB(n) is greater than 2*FCB ave A negative determination at block 804 may cause the logic to proceed to block 806, where TFB(n) = TFB(n). However, a positive determination at block 804 may instead cause the logic to proceed to block 808, where TFB(n) = TargetBitAllocate(n).

[0063] 9, which illustrates exemplary overall logic for a rate control algorithm consistent with the present principles. Beginning at block 900, a device may load a new frame and then determine whether the frame is an I-frame at diamond-shaped decision block 902. A negative determination at decision block 902 may cause the logic to proceed to block 904, where the device may use an RD (rate-distortion to maintain video quality) model to calculate a QP and update the rate buffer.

[0064] However, in response to a positive determination at decision block 902, the logic may instead proceed to block 906, where the device performs a target bit allocation, and may proceed to block 908, where the device obtains the output PIDCB(n) from the feedback loop 700 of Figure 7, and determines the target frame bits TFB(n), at block 910. The logic may then proceed to block 912, where the device may perform the TFB(n) update algorithm of Figure 8.

[0065] The logic may then proceed to block 904, where the device may use the RD model to calculate the QP and update the rate buffer. From block 904, the logic may then proceed to block 914, where the device may obtain the resulting PID error PIDE(n) and feed PIDE(n) back as an input to the loop 700.

[0066] Thus, a rate buffer may be maintained at the encoder device and / or decoder device using the logic of FIG.

[0067] Although particular embodiments have been shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. 1. An apparatus including at least one processor, the at least one processor comprising: configured to maintain at least one video buffer according to a rate buffer bit controller feedback loop and a proportional / integral / derivative (PID) bit controller feedback loop, the PID bit controller feedback loop comprising:

1. An apparatus comprising: a first branch, a second branch, and a third branch in parallel with one another, each branch receiving a feedback error, the first branch configured to modify a target bit rate based at least in part on the feedback error over a first period of time, the second branch configured to modify the target bit rate based at least in part on the feedback error over a second period of time that is longer than the first period of time, and the third branch configured to reduce overshoot.

2. The at least one processor: configured to use a predefined first quantization parameter (QP) for a first I-frame of a group of pictures (GOP), the first I-frame being the first I-frame of the GOP; 2. The apparatus of claim 1, wherein the apparatus is configured to use a second QP for a second I-frame of the GOP, the second QP being determined by the at least one processor from a third QP of a previous I-frame in the GOP and a fourth QP of a last non-I-frame of the GOP that preceded the second I-frame.

3. The apparatus of claim 2 , wherein the previous I-frame is the last I-frame in the GOP that preceded the second I-frame.

4. The apparatus of claim 2 , wherein the second QP is determined from an average of the third QP and the fourth QP.

5. The apparatus of claim 2 , wherein the second I-frame is a frame from the same scene as the first I-frame.

6. the same scene is a first scene, and the at least one processor:

6. The apparatus of claim 5, further comprising: a third QP for a third I-frame of the GOP, the third I-frame being a frame for a second scene different from the first scene; the third QP being determined by the at least one processor from an average of QPs of two or more preceding I-frames from the GOP; and the two or more QPs of the preceding I-frames being associated with the first scene.

7. The apparatus of claim 2 , wherein the third QP is different from the first QP.

8. The processor: The apparatus of claim 1 , configured to determine a quantization parameter (QP) for a non-I frame based on performance of the rate buffer bit controller feedback loop and the PID bit controller feedback loop.

9. The first branch adjusts a first constant K to modify the target bit rate based at least in part on the feedback error over the first time period. Ｐ and the second branch uses a second constant K to modify the target bit rate based at least in part on the feedback error over the second time period. Ｉ and the third branch uses a third constant K to reduce overshoot. Ｄ 2. The apparatus of claim 1, wherein

10. 10. The apparatus of claim 9, wherein the first constant, the second constant, and the third constant are non-negative numbers determined using a machine learning based model.

11. The first branch is a function K Ｐ (PIDE(n)), and the second branch is [Equation 1] and the third branch is applied to the function K Ｄ 10. The apparatus of claim 9, wherein the apparatus applies (dPIDE(n) / dn).

12. The apparatus of claim 1 , further comprising an encoder device, wherein the at least one video buffer is maintained by the at least one processor in the encoder device.

13. The apparatus of claim 1 , further comprising a decoder device, wherein the at least one video buffer is maintained by the at least one processor in the decoder device.

14. Implements a rate buffer bit controller feedback loop, Implements a proportional / integral / derivative (PID) bit controller feedback loop, The method comprises maintaining at least one video buffer according to the rate buffer bit controller feedback loop and the proportional / integral / derivative (PID) bit controller feedback loop.

15. processing a first feedback error over a first time period via a first branch of the PID bit controller feedback loop to modify a target bit rate; processing a second feedback error over a second time period via a second branch of the PID bit controller feedback loop to modify the target bit rate, the second time period being longer than the first time period; 15. The method of claim 14, further comprising processing a third feedback error via a third branch of the PID bit controller feedback loop to reduce overshoot.

16. The first branch adjusts a first constant K to modify the target bit rate based at least in part on the first feedback error over the first time period. Ｐ and a function (PIDE(n)), wherein the second branch uses a second constant K to modify the target bit rate based at least in part on the second feedback error over the second time period. Ｉ and functions [Equation 2] and the third branch uses a third constant K to reduce overshoot. Ｄ and function K Ｄ 16. The method of claim 15, wherein (dPIDE(n) / dn) is used.

17. using a predefined first quantization parameter (QP) for a first I-frame of a group of pictures (GOP), the first I-frame being the first I-frame of the GOP; 15. The method of claim 14, using a second QP for a second I-frame of the GOP, the second QP being determined from an average of a third QP of a previous I-frame in the GOP and a fourth QP of a last non-I-frame of the GOP that preceded the second I-frame.

18. 1. A device including at least one computer storage, the at least one computer storage being configured to store non-transitory signals; instructions executable by at least one processor for using a proportional / integral / derivative (PID) bit controller feedback loop to maintain at least one video buffer, the PID bit controller feedback loop comprising:

1. A device comprising: a first branch, a second branch, and a third branch in parallel with one another, each receiving a feedback error, the first branch configured to modify a target bit rate based at least in part on the feedback error over a first period of time, the second branch configured to modify the target bit rate based at least in part on the feedback error over a second period of time that is longer than the first period of time, and the third branch configured to reduce overshoot.

19. The instruction:

20. The device of claim 18, wherein the device is executable to use a rate buffer bit controller feedback loop and the PID bit controller feedback loop to maintain the at least one video buffer.

20. The first branch adjusts a first constant K to modify the target bit rate based at least in part on the feedback error over the first time period. Ｐ and a function (PIDE(n)), wherein the second branch uses a second constant K to modify the target bit rate based at least in part on the feedback error over the second time period. Ｉ and functions [Equation 3] and the third branch uses a third constant K to reduce overshoot. Ｄ and function K Ｄ 20. The device of claim 18, using (dPIDE(n) / dn).

Images