Encoding rate control of real-time videos using dynamic resolution switching

Abstract

Method (700) for encoding rate control of real-time videos, comprising: Identifying the first frame of a multitude of frames in a video data stream; the encoding (704) of the first frame at a first bit rate (108) that is representative of a quantification parameter (106); comparing (710) the first bit rate (108) with a target bit rate (104) for the first frame and comparing (712; 718) the quantification parameter (106) with either a minimum quantification threshold (100) or a maximum quantification threshold (102), wherein If the first bit rate (108) is greater than the target bit rate (104), the quantification parameter (106) is compared with the maximum quantification threshold (102), and if the first bit rate (108) is less than the target bit rate (104), the quantification parameter (106) is compared with the minimum quantification threshold (100); the adjustment (714; 716; 720; 722) of at least one encoding parameter in response to the comparison (710) of the first bit rate (108) with the target bit rate (104) and the comparison (712; 718) of the quantification parameter (106) with either the minimum quantification threshold (100) or the maximum quantification threshold (102); Identifying a second frame from a multitude of frames in the video data stream; encoding the second frame at a second bitrate using the adapted at least one encoding parameter; and the transmission of the encoded first frame and the encoded second frame in an output bitstream.

Description

BACKGROUND

[0001] Digital video streams typically represent videos using a sequence of frames or still image files. Each frame may contain a number of blocks, which in turn may include information describing the value of color, brightness, or other pixel attributes. The data volume of a typical video stream is large, and the transmission and storage of video can correspondingly require significant computing and communication resources. Due to the large amount of data associated with video data, a powerful compression method is required for both its transmission and storage. Furthermore, it is desirable to be able to stream high-quality video data at a resolution optimized for the available connections.

[0002] MIYATA, Kouji; TAGUCHI, Akira: A novel MPEG-4 rate control method with spatial resolution conversion for low bit-rate coding. In: IEEE International Symposium on Circuits and Systems (ISCAS) - Conference proceedings, May 23-26, pp. 4538-4541, refers to a rate control method for MPEG-4.

[0003] US 2012 / 0 294 355 A1 describes an apparatus and a method for transcoding a compressed video stream. SUMMARY

[0004] Disclosed herein are aspects of systems, methods, and apparatus for employing dynamic resolution switching to achieve a target bit rate for encoding a video data stream.

[0005] In a first aspect, a method according to claims 1 to 7 is disclosed.

[0006] In another aspect, a device according to claims 8 and 9 is disclosed.

[0007] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims, and the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The invention will best be understood from the following detailed description when read in conjunction with the accompanying drawings. Unless otherwise indicated, like reference numerals refer to like elements throughout the various figures. Fig. Figure 1A is a graph showing a target bit rate for encoding a video stream that is exceeded due to a maximum quantization parameter. Fig. Figure 1B is a graph showing a target bit rate for encoding a video stream that is exceeded due to a minimum quantization parameter. Fig. Figure 2 is a circuit diagram of a video encoding and decoding system. Fig. 3 is a block diagram of an example of a computing device that may implement a transmitting station or a receiving station. Fig. Figure 4 is a diagram of a typical video stream to be encoded and then decoded. Fig. 5 is a block diagram of a video compression system according to one aspect of the teachings herein. Fig. 6 is a block diagram of a video decompression system according to another aspect of the teachings herein. Fig. 7 is a flowchart of an example of a process for achieving a target encoding bit rate using single-pass encoding. Fig. 8 is a flowchart of an example of a process for achieving a target encoding bit rate using two-pass encoding. DETAILED DESCRIPTION

[0009] Rate control algorithms in video coding devices aim to maintain a constant bit rate generated by the code when the complexity of the input data varies over time. A constant bit rate can be achieved by adjusting a quantization parameter (QP) for each frame based on the amount of bits generated by the previous frame(s). In general, higher quantization reduces the bit rate, and lower quantization increases it. The quantization parameter indicates the order of magnitude at which data of a video stream is quantized by a coding device. Consequently, in cases where a higher quantization parameter is used by a coding device, the quantization stage involves a higher reduction of the video data for higher quantization parameters and a lower reduction of the video data for lower quantification parameters.The resulting quantized video data indicates a bitrate at which a frame of a video stream containing the data is encoded. In general, using a higher quantization parameter for a given frame of a video stream would result in a lower frame bitrate, and using a lower quantization parameter would result in a higher frame bitrate.

[0010] Nevertheless, depending on the complexity of the video sequence and the varying capabilities of the encoding device, there may be cases where a target bitrate for encoding a given video stream cannot be achieved. This may be due to the QP having already been limited to a maximum threshold (e.g., a QP value whose increase would result in a reduction in video quality relative to the resolution by producing obvious visual artifacts) and yet still having a bitrate that is not low enough, thus resulting in a bitrate overshoot. Alternatively, the use of a large QP may lead to encoding artifacts that have a detrimental impact on video quality. Similarly, if the desire is to optimize video quality for a user at a given bandwidth and the QP has reached a minimum threshold (e.g.,If you have a QP value for which further reductions have no effect on improving video quality relative to resolution, there may be no other way to further improve video quality. Consequently, the user experience suffers.

[0011] Fig. 1A and Fig. 1B are graphs each illustrating a target bit rate for encoding a video stream, which has been exceeded due to a maximum quantization parameter and which has been undershot due to a minimum quantization parameter. The horizontal axes indicate the sequential progression of the frames of a video stream encoded by an encoding device, and the vertical axes indicate a quantization parameter used at a stage of the encoding device for the respective frame. The minimum QP threshold 100 and the maximum QP threshold 102 for the encoded frames of the video stream are represented as constant values ​​relative to the target bit rate 104. The figures further indicate that, depending on the data being encoded, the target bit rate 104 may not be achievable solely by increasing or decreasing the QP 106.The frame bit rate 108 may vary, for example, due to the complexity of encoding the input data comprising a given frame.

[0012] As in Fig. 1A, the frame bit rate 108 increases as the data of the video stream becomes more complex. In response to an increase in the frame bit rate 108, the encoding device increases the QP 106 while attempting to maintain the target bit rate 104 for the video stream. Eventually, however, the QP 106 reaches the maximum QP threshold 102 and thus cannot be increased any further, even though the frame bit rate 108 has still not reached the target bit rate 104, because the encoded data is still too large to achieve the target bit rate 104. This is referred to as a bit rate overshoot because the frame bit rate 108 is still above the target bit rate 104 despite the fact that the maximum QP threshold 102 is limited. As can be seen in a similar manner from Fig. As can be seen from Figure 1B, the coding device is capable of lowering the QP 106 if it is determined that the target bit rate 104 could not be achieved due to the fact that the frame bit rate 108 is too low. If the frame bit rate 108 remains below the target bit rate 104 despite the fact that the minimum QP threshold 100 has been achieved, this results in a bit rate undershoot.

[0013] Accordingly, conventional coding devices are not capable of achieving a constant bit rate for encoding video data by simply adjusting a QP. While some coding devices are further capable of inserting a key frame for encoding, this solution is not optimal, at least due to the fact that it necessarily leads to a bit rate peak when the key frame is encoded. In contrast, the present disclosure describes dynamic scaling of the reference frame to improve the implementation of rate control by switching the resolution during encoding. Rate control is thus achieved, for example, by the coding device by means of internal scaling of the reference frame prior to forming the prediction signal intended for it. Single-pass and two-pass coding devices for encoding a video data stream are described.In a two-pass encoding system, a first pass encodes frames of a video stream using a constant quantizer, and a second pass assigns bits to each frame in an optimal manner, knowing in advance which frames are complex and which frames are simple. Further details of dynamic scaling of the reference frame are described herein, with initial reference to a system in which it can be implemented.

[0014] Fig. Figure 2 is a circuit diagram of a video encoding and decoding system 200. A transmission station 202 may be, for example, a computer having an internal configuration of hardware such as that shown in Fig. 3. However, other suitable implementations of the transmission station 202 are possible. For example, the processing of the data from the transmission station 202 can be distributed among multiple devices.

[0015] A network 204 may connect the transmitting station 202 to a receiving station 206 for encoding and decoding the video stream. Specifically, the video stream may be encoded in the transmitting station 202, and the encoded video stream may be decoded in the receiving station 206. Network 204 may be, for example, the Internet. Network 204 may also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular network, or any other means for transmitting the video stream from the transmitting station 202 to the receiving station 206, as illustrated in this specific example.

[0016] In one example, the receiving station 206 may be a computer having an internal configuration of hardware, such as that described in Fig. 3. However, other suitable implementations of the receiving station 206 are possible. For example, the processing of the data from the receiving station 206 can be distributed among multiple devices.

[0017] Other implementations of video encoding and decoding system 200 are possible. For example, one implementation may omit network 204. In another implementation, a video stream may be encoded and then stored for transmission at a later time to receiving station 206 or any other device having memory. In one implementation, receiving station 206 receives (e.g., via network 204, a computer bus, and / or some communication path) the encoded video stream and stores the video stream for later decoding. In one exemplary implementation, a real-time transport protocol (RTP) is used to transmit the encoded video over network 204. In another implementation, a transport protocol that does not include RTP may be used, such as an HTTP-based video streaming protocol.

[0018] For example, when used in a video conferencing system, transmitting station 202 and / or receiving station 206 may have the capability to both encode and decode a video stream, as described below. For example, receiving station 206 may be a video conferencing participant who receives an encoded video bitstream from a video conferencing server (e.g., transmitting station 202) for decoding and viewing, and further transmits its own video bitstream to the video conferencing server for decoded and viewing by other participants.

[0019] Fig. 3 is a block diagram of an example of a computing device 300 that may implement a transmitting station or a receiving station. For example, computing device 300 may implement both transmitting station 202 and receiving station 206 of Fig. 2. The computing device 300 may be available in the form of a computing system, including multiple computing devices, or in the form of a single computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.

[0020] A CPU 302 in computing device 300 may be a conventional central processing unit. Alternatively, CPU 302 may be any other type of device, or multiple devices, capable of manipulating and processing information that does not yet exist or is developed at a later date. Although the disclosed implementations may be performed with a single processor, as will be appreciated, such as CPU 302, speed and efficiency benefits may be achieved using more than one processor.

[0021] Memory 304 in computing device 300 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of memory device may be used as memory 304. Memory 304 may include code and data 306 accessible by CPU 302 using bus 312. Memory 304 may further include an operating system 308 and application programs 310, where application programs 310 include the at least one program that grants permission to CPU 302 to perform the methods described herein. For example, application programs 310 may include applications 1 through N, which may further include a video encoding application that performs the methods described herein.Computing device 300 may also include a second storage device 314, which may be, for example, a memory card used with a mobile computing device. Since the video communication sessions may contain a significant amount of information, they may be stored in whole or in part in secondary storage 314 and loaded onto storage 304 when needed for further processing.

[0022] Computing device 300 may also include one or more output devices, such as a display 318. In one example, display 318 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input during operation. Display 318 may be coupled to CPU 302 via bus 312. Other output devices that allow a user to program and otherwise use computing device 300 may be provided in addition to, or as an alternative to, display 318. If the output device is or includes a display, the display may be implemented in a variety of ways, including a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light-emitting diode (LED) display, such as an OLED display.

[0023] Computing device 300 may also include or be in communication with an image sensing device 320, such as a camera, or any other existing or later developed image sensing device 320 that can sense an image, such as an image of a user operating computing device 300. Image sensing device 320 may be positioned to directly face the user operating computing device 300. In one example, the position and optical axis of image sensing device 320 may be configured such that the field of view includes an area directly adjacent to display 318 and from which display 318 is visible.

[0024] Computing device 300 may also include a sound sensing device 322, such as a microphone or any other sound sensing device now existing or later developed, capable of sensing, or communicating with, sounds in the vicinity of computing device 300. Sound sensing device 322 may be positioned to face the user operating computing device 300 and may be configured to receive sounds, such as speech sounds or other utterance-related sounds, made by the user while the user is operating computing device 300.

[0025] Although the CPU 302 and memory 304 of the computing device 300 in Fig. 3, as integrated within a single unit, other configurations may also be used. The operations of CPU 302 may be distributed across multiple machines (each machine having one or more processors), which may be coupled directly or via a local area network or other network. Memory 304 may be distributed across multiple machines, such as network-based storage or storage within multiple machines that perform the operations of computing device 300. Although bus 312 of computing device 300 is depicted here as a single bus, it may consist of multiple buses. Furthermore, secondary storage 314 may be directly coupled to the other components of computing device 300 or may be accessible via a network and may comprise a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards.The computing device 300 can thus be implemented in a wide variety of configurations.

[0026] Fig. 4 is a diagram of an example of a video stream 400 to be encoded and then subsequently decoded. The video stream 400 includes a video sequence 402. At the next level, the video sequence 402 includes a number of contiguous frames 404. While three frames are mapped as the contiguous frames 404, the video sequence 402 may include any number of contiguous frames 404. The contiguous frames 404 may then be further divided into individual frames, e.g., a frame 406. At the next level, the frame 406 may be divided into a series of segments or levels 408. The segments 408 may be subsets of frames, allowing, for example, parallel processing. The segments 408 may also be subsets of frames that may separate the video data into different colors. For example, a frame 406 of color video data may include one luminance plane and two chrominance planes.The segments 408 can be sampled at different resolutions.

[0027] Regardless of whether the frame 406 is divided into segments 408 or not, the frame 406 may be further divided into blocks 410, which include data corresponding to, for example, 16×16 pixels within the frame 406. The blocks 410 may also be arranged to contain data from one or more levels 408 of pixel data. The blocks 410 may also have any other suitable size, such as 4×4 pixels, 8×8 pixels, 16×8 pixels, 8×16 pixels, 16×16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

[0028] Fig. 5 is a block diagram of an encoding device 500 according to one implementation. The encoding device 500 may be implemented in the transmission station 202, as described above, such as by providing a computer software program stored in memory, such as memory 304. The computer software program may include machine instructions that, when executed by a processor, such as CPU 302, cause the transmission station 202 to encode video data to the Fig. 5. The coding device 500 can also be implemented as specialized hardware, for example, forming part of the transmission station 202. In a particularly desirable implementation, the coding device 500 is a hardware coding device. The coding device 500 has the following stages to perform the various functions in a forward path (represented by the solid connecting lines) to create an encoded or compressed bitstream 520 using the video data stream 400 as input: an intra / inter prediction stage 502, a transformation stage 504, a quantization stage 506, and an entropy coding stage 508. The coding device 500 can also include a reconstruction path (represented by the dotted connecting line) to reconstruct a frame for encoding future blocks. In Fig. 5, the coding device 500 has the following stages to perform the various functions in the reconstruction path: a dequantization stage 510, an inverse transform stage 512, a reconstruction stage 514, and a loop filtering stage 516. Other structural variations of the coding device 500 may be used to encode the video data stream 400.

[0029] When the video data stream 400 is presented for encoding, each frame 406 may be processed in block units. In the intra / inter prediction stage 502, each block may be encoded using intra-frame prediction (also called intra-prediction) or inter-frame prediction (also called inter-prediction). In either case, a prediction block may be formed. In the case of intra-prediction, a prediction block may be formed from samples of the current frame that have been previously encoded and reconstructed. In the case of intra-prediction, a prediction block may be formed from samples located in one or more of the previously constructed reference frames.

[0030] Thereafter, and still with reference to Fig. 5, the prediction block may be subtracted from the current block in the intra / inter-prediction stage 502 to generate a residual block (also called a residue). The transformation stage 504 converts the residue into transform coefficients—for example, into the frequency domain using block-based transforms. The quantization stage 506 converts the transform coefficients into discrete quantum values, referred to as quantized transform coefficients, using a value or quantization level output by the quantizer. For example, the transform coefficients may be divided by the value output by the quantizer and truncated. The quantized transform coefficients are then entropy encoded in the entropy encoding stage 508.The entropy-encoded coefficients are then output to the compressed bitstream 520 along with other information used to decode the block, including, for example, the type of prediction used, the transform type, motion vectors, and the value output by the quantizer. The compressed bitstream 520 may be formatted using various techniques, such as variable-length coding (VLC) or arithmetic coding. The compressed bitstream 520 may also be referred to as an encoded video data stream or an encoded video bitstream, and the terms may be used interchangeably herein.

[0031] The reconstruction path in Fig. 5 (represented by the dotted connecting line) can be used to ensure that both the encoding device 500 and the decoding device 600 (described below) use the same reference frame to decode the compressed bitstream 520. The reconstruction path performs functions that have similar properties to functions that take place during the decoding process, which will be discussed in more detail below, including the dequantization of the quantized transform coefficients in the dequantization stage 510 and the inverse transformation of the dequantized transform coefficients in the inverse transformation stage 512 to generate a derivative residual block (also referred to as a derivative residue).In reconstruction stage 414, the prediction block predicted in intra / inter prediction stage 502 may be added to the derivative residual to generate a reconstructed block. Loop filtering stage 516 may be applied to the reconstructed block to reduce distortion, such as blocking artifacts.

[0032] Other variations of the encoding device 500 may be used to encode the compressed bitstream 520. For example, a non-transform-based encoding device may quantize the remaining signal directly for specific blocks or frames without the transform stage 504. In another implementation, the encoding device may have a quantization stage 506 and a dequantization stage 510 combined into a single stage.

[0033] Fig. 6 is a block diagram of a decoding device 600 in accordance with another implementation. The decoding device 600 may be implemented in the receiving station 206, for example, by providing a computer software program stored in memory 304. The computer software program may include machine instructions that, when executed by a processor, such as a CPU 302, cause the receiving station 206 to process video data in the manner described in Fig. 6. The decoding device 600 may also be implemented in hardware included, for example, in the transmitting station 202 or the receiving station 206.

[0034] The decoder 600, which has similar characteristics to the reconstruction path of the encoding device 500 discussed above, includes an example of the following stages to perform various functions to generate an output video data stream 616 from the compressed bitstream 520: an entropy decoding stage 602, a dequantization stage 604, an inverse transform stage 606, an intra / inter prediction stage 608, a reconstruction stage 610, a loop filtering stage 612, and a deblocking filtering stage 614. Other structural variations of the decoder 600 may be used to decode the compressed bitstream 520.

[0035] When the compressed bitstream 520 is presented for decoding, data elements within the compressed bitstream 520 may be decoded by the entropy decoding stage 602 to generate a set of quantized transform coefficients. In the dequantization stage 604, the quantized transform coefficients are dequantized (e.g., by multiplying the quantized transform coefficients by the value output by the quantizer), and in the inverse transformation stage 606, the dequantized transform coefficients are inversely transformed to generate a derivative residue identical to the residue generated in the inverse transformation stage 512 in the encoder 500.Using the header information decoded from the compressed bitstream 520, the decoding device 600 may use the intra / inter prediction stage 608 to generate the same prediction block that was generated in the encoding device 500, e.g., in the intra / inter prediction stage 502. In the reconstruction stage 610, the prediction block may be added to the derivative residue to generate a reconstructed block. The loop filtering stage 612 may be applied to the reconstructed block to reduce blocking artifacts. Other filtering operations may be applied to the reconstructed block. In this example, the deblocking filtering stage 614 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 616.The output video stream 616 may also be referred to as a decoded video stream, and the terms may be used interchangeably. Other variations of the decoding device 600 may be used to decode the compressed bitstream 520. For example, the decoding device 600 may generate the output video stream 616 without the deblocking filtering stage 614.

[0036] Both Fig. 7 and Fig. 8 are flowcharts of methods 700 and 800 for using resolution scaling to achieve a constant target bit rate in single-pass and two-pass encoding, respectively. Methods 700 and 800 may be implemented in a system, such as computing device 300, to support the encoding of a video data stream. For example, processes 700 and 800 may be implemented as a software program executed by a computing device, such as transmitting station 202 or receiving station 206. The software program may include machine-readable instructions stored in a memory, such as memory 304, that, when executed by a processor, such as CPU 302, causes the computing device to perform either method 700 or method 800. Methods 700 and 800 may also be implemented, in whole or in part, using hardware.As explained above, some computing devices may have multiple memories and multiple processors, and the steps or operations of each of methods 700 and 800 may, in such cases, be distributed using different processors and memories. The use of the terms "processor" and "memory" in the singular herein encompasses both computing devices having only one processor or memory, and computing devices having multiple processors or memories, each used in performing some, but not necessarily all, of the specified steps.

[0037] For ease of explanation, both method 700 and method 800 are illustrated and described as a series of steps or operations. However, steps and operations consistent with this disclosure may occur in different orders and / or concurrently. Moreover, steps or operations consistent with this disclosure may occur in conjunction with other steps or operations not presented or described herein. Furthermore, not all illustrated steps or operations are strictly necessary to implement a method according to the disclosed subject matter. One of method 700 or 800 may be repeated for each frame of the input signal.

[0038] With reference to Fig. 7, the method 700 for using in-frame resolution scaling of a single-pass encoding to achieve a relatively constant target bit rate is initiated by setting, at operation 702, values ​​for a target bit rate, an initial QP, a minimum QP threshold, and a maximum QP threshold. These values ​​may be indicative of an initial resolution at which the video stream will be encoded. At operation 704, an encoding device encodes a first frame of a video stream using the initial QP. If the frame-bit-rate encoding of the first frame results in the target bit rate, operation 706 prepares the encoding device for encoding the next frame of the video stream via return to operation 704.Otherwise, method 700 proceeds to a trigger adjustment phase of operation 708, which uses filtering logic to determine whether oscillations are present in the video data. If no oscillations are present, operation 708 prepares the encoder to encode the next frame of the video stream by returning to operation 704. Otherwise, method 700 proceeds to an adjustment phase initiated at operation 710.

[0039] At operation 710, it is determined whether an overshoot or an undershoot is the reason why the frame bit rate does not match the target bit rate. An overshoot occurs in cases where the frame bit rate exceeds (exceeds) the target bit rate, for example, because the QP is too low for the resolution at which the video stream was encoded. An undershoot occurs in cases where the frame bit rate falls below (undershoots) the target bit rate, for example, because the QP is too high for the resolution at which the video stream was encoded. Regardless of whether an overshoot or an undershoot is determined, one or more encoding parameters of the video stream are subsequently adjusted to improve the quality of the video stream.Encoding parameters can include any value that, when adjusted, affects the quality or visual presentation of the video stream. In one implementation, encoding parameters include, without limitation, the QP, the video resolution, the maximum QP threshold, and the minimum QP threshold. The subsequent adjustments made by the encoder depend on whether the bitrate issue is due to an overshoot or undershoot.

[0040] If operation 710 determines that a bit rate overshoot has occurred, method 700 proceeds to operation 712, where it is determined whether the value currently set for the QP (which may be the initial QP set at operation 702 in cases where operation 710 has been performed only once) equals the value currently set for the maximum QP threshold. If not, the QP is increased at operation 714, and this operation prepares the encoder to encode the next frame of the video stream using the increased QP by returning to operation 704. However, if the value of the QP equals the maximum QP threshold, the QP cannot be increased any further because it has already reached the most effective value relative to the resolution of the video stream.Accordingly, method 700 lowers the resolution of the video stream at operation 716, which involves readjusting the values ​​of the minimum QP threshold and the maximum QP threshold. Operation 716 also lowers the QP value based on the reduction in resolution. Operation 716 then prepares the encoder to encode the next frame using the new values ​​of the resolution and the new QP, the minimum QP threshold, and the maximum QP threshold by returning to operation 704.

[0041] If operation 710 instead determines that a bit rate undershoot is present, method 700 proceeds to operation 718, where it is determined that the value currently set for the QP (which may be the initial QP set at operation 702 in cases where operation 710 was performed only once) equals the value currently set for the minimum QP threshold. If not, the QP is decremented at operation 720, and this operation prepares the encoder to encode the next frame of the video stream using the decremented QP by returning to operation 704.However, if the QP value equals the minimum QP threshold value, the QP cannot be decreased any further because it has already reached the least effective value relative to the resolution of the video stream. Accordingly, method 700 increases the resolution of the video stream at operation 722, which involves readjusting the values ​​of the minimum QP threshold and the maximum QP threshold. Operation 722 also increases the QP value based on the increase in resolution. Operation 722 then prepares the encoder to encode the next frame using the new values ​​of the resolution and the new QP, the minimum QP threshold, and the maximum QP threshold by returning to operation 704.

[0042] If a higher-resolution source is available, another option is to increase the video resolution where the encoding bitrate consistently falls below the target bitrate, or where the encoding bitrate exceeds the target bitrate above a certain threshold number of times. For example, in cases where a user chooses to watch a video at 360p resolution but their bandwidth allows viewing at a higher resolution, the video resolution can be increased to a maximum resolution at which the user can stream the video over their connection.

[0043] In two-pass coding, a first pass of encoding frames of a video stream involves determining which frames are too difficult or too easy to encode based on a quantization parameter, for example, by detecting a number of bits generated by encoding an input frame corresponding to a first frame of the video stream. If the generated number of bits indicates that the input frame is too difficult or too easy to encode, the encoding device can adjust the resolution of the video stream after a second pass of encoding the corresponding frame. In this way, two-pass encoding can prevent most undershoots and overshoots by identifying potential instances of undershoots and overshoots during a first pass and making the necessary adjustments to effectively encode the video stream in a second pass.The amount of undershoot and overshoot can indicate the degree to which the resolution per dimension needs to be adjusted.

[0044] If we now refer to Fig. 8, the method 800 for using resolution scaling in two-pass encoding to achieve a relatively constant target bit rate starts at operation 802 by setting initial values ​​for a target bit rate and a QP. At operation 804, an input frame is received and encoded using a set target bit rate and QP values. In response to encoding the input frame, a number of bits is generated at operation 806. The number of bits is indicative of a bit rate at which the input frame will be encoded. Depending on the size of the generated bits, the method 800 may proceed to an adjustment phase for adjusting the encoding parameters of the video stream, such as the resolution of a video.In one implementation, the decision about whether to proceed to the next adjustment phase may be based on whether the frame bit rate falls outside a specified range, within which QP results vary with respect to the desirable video quality (e.g., video streaming without visible encoding artifacts relative to the resolution). For example, if the frame bit rate is within 100 to 200 kbps of the target bit rate, operation 808 may make the decision not to proceed to the adjustment phase. In the event that no adjustments need to be made, method 800 proceeds to operation 816, where an encoded frame is generated based on the previously adjusted OP value, and this encoded frame is then transmitted as part of the output video stream at operation 818. In one implementation, the setting at operation 808 may be encoded using a flag.

[0045] If it is determined at operation 808 that an adjustment to the video resolution needs to be made, method 800 proceeds to operation 810, which determines whether the need for the adjustment is due to the fact that the input frame encoding bitrate either falls below or exceeds the target bitrate. If it exceeds the target bitrate, one or more encoding parameters are adjusted at operation 812, for example, by reducing the resolution of the video stream. If it falls below the target bitrate, one or more encoding parameters are adjusted at operation 814, for example, by increasing the resolution in the video stream.

[0046] In one implementation, the resolution adjustment of the video may be performed using a square root of the weighted inverse of the percentage bitrate deviation, such as S new = √(1 / (α*B dev )), where S newrepresents the new scaling per dimension ratio of the resolution, a represents an a-weighting parameter and B dev represents a deviation from the bitrate range. For example, in cases where encoding the input frame results in less than a 20% overshoot and the weighted parameter is set to 1, the input frame of the video stream has a resolution of 1280 × 720, perhaps around S new = √(1 / (1*1,2)) = 0.913, resulting in a new streaming resolution of 1168 × 657. Similarly, in those cases where encoding an input frame results in a serious undershoot of 50%, the scaling ratio can be S new= √(1 / (1*0.5)) = 1.414. In this way, the resolution of the video stream can be adjusted to be encoded at 1810 × 1018. Adjusting the weighting parameter a above 1 can result in a new frame bitrate closer to a midpoint in terms of the acceptable spectrum (e.g., 150 kbps for a spectrum of 100 to 200 kbps).

[0047] After adjusting the video resolution, method 800 proceeds to operation 816, where an encoded frame corresponding to an input frame is encoded as part of a second pass of a video stream. The encoded frame is therefore transmitted as part of the output video stream at operation 818.

[0048] Resolution scaling can provide additional benefits beyond those described above. For example, resolution scaling can improve the performance of encoders that have not found an acceptable motion vector for inter-prediction with an established motion search window. This typically results in poor compression due to overuse of intra-prediction. By downscaling the video stream in such a situation, the bit rate is reduced because there are fewer pixels to encode and the amount of absolute motion in the video stream is also reduced. This can improve the use of the search window.

[0049] In one implementation, and in response to determining an undershoot at operation 810, operation 814 includes decreasing the size of the motion search area, which may also be performed instead of or in addition to decreasing the resolution of the video. Similarly, in response to determining an overshoot at operation 810, operation 812 includes increasing the size of the motion search area instead of or in addition to increasing the resolution of the video. In one implementation, changes to the size of the motion search area are made only for certain color components of the input frame, for example, the luminance and / or chrominance pixel values ​​thereof.

[0050] In one implementation, one or more encoder constraints related to the motion search range may be identified, for example, in response to encoding the input frame at operation 804. The encoder constraints may indicate instances where a motion vector found by encoder 500 exceeds a threshold, for example, in those cases where the video stream includes fast-moving scenes, including an amount of motion that exceeds the ability of encoder 500 to adequately represent the scenes at a given resolution. In one implementation, the resolution of the video stream may be reduced in response to the encoder constraints indicating that the motion vector exceeds the threshold.

[0051] In one implementation, the size of the motion search area is not subject to any changes. In cases where the encoding device 500 is implemented, for example, as a hardware encoding device, the motion search area can be statistically set to a size of N × M, where N and M have either the same value or different values. The motion search area can therefore only cover a portion of an input frame if the resolution of the video data stream is higher than N × M. A center location of the motion search area can be defined, for example, with respect to an initial reference location of an initial portion of the input frame to be covered (e.g., a center location of a current block or slice to be encoded).However, in one implementation, the resolution of the video data stream may be reduced, for example, to improve the ability of the encoding device 500 to detect motion within the motion search region by causing the motion search region to cover a larger portion of the input frame. The decision to reduce the resolution in this manner (e.g., as part of determining whether to adjust the encoding parameters at operation 808) may be made, as described above, for example, based on a threshold or other encoding device constraint relative to the motion search region. The reduction in resolution causes the center point location of the motion search region to be readjusted based on a reference location of the portion of the input frame that is consequently covered by the motion search region.Similarly, in one implementation, the resolution of the video stream may be increased based on a threshold or other encoder constraint. For example, in cases where the resolution is less than the size of the motion search region (e.g., in cases where the motion search region covers more than an entire input frame), the encoder 500 may increase the resolution of the stream to match the size of the motion search region. Therefore, the center location of the motion search region could be located at, or remain at, the reference location of the input frame. As used herein, the term "center" refers to a central position relative to corresponding dimensions.

[0052] Although the one or more encoder limitations on the motion search range are described herein with respect to the two-pass coding of Fig. 8, they also apply in the context of a single-pass coding, such as that described with reference to Fig. 7. For example, adjusting the encoding parameters of a first frame of the video stream may involve identifying encoder constraints with respect to a motion search region (e.g., as part of determining a trigger adjustment at operation 708) or decreasing (or increasing) the resolution of the video stream based on the encoder constraints. In one implementation, changing the resolution in this manner results in readjusting a center location of the motion search region with respect to a reference location of the portion of the first frame that is consequently covered by the motion search region.

[0053] In another implementation, where the encoder 500 is a hardware encoder, the motion search parameters remain the same from block to block and frame to frame, except that the center location of the motion search area changes based on the block of the frame being encoded. Therefore, more or less motion may be detected by changing the resolution. For example, changing the resolution for certain blocks within the frame may provide a better match and thus lower residuals when fast motion is present, since the lower resolution detects more motion within the same search area.

[0054] The target bitrate set in the frame of the initial operations of processes 700 and 800 may be a variable bitrate subject to fluctuations. These fluctuations may occur as a result of a change in the connection speed or network connection type of a user streaming a video. For example, in cases where a user initially streams a video over a Wi-Fi connection that suddenly becomes unavailable, the video stream may be switched to an LTE connection. Due to the fact that each network connection type may infer its own target bitrate for optimal viewing of a given video stream at a given resolution, it may be desirable to use implementations that handle the disclosed rate control in frame of single-pass encoding and two-pass encoding for variable target bitrates.

[0055] The encoding and decoding aspects described above illustrate some examples of encoding and decoding techniques. However, it is understood that encoding and decoding, as used in the claims, could mean compression, decompression, transformation, or any other processing or modification of data.

[0056] As used herein, the word "example" means serving as an example, exemplar, or illustration. Any aspect or design described herein as an "example" is not necessarily to be construed as preferable or advantageous over other aspects or designs. The intent of using the word "example" is to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" and not an exclusive "or." That is, unless otherwise stated or clear from the context, the phrase "X includes A or B" is intended to mean all of the natural inclusive permutations. That is, if X includes A; X includes B; or X includes both A and B, then, among all of the foregoing instances, "X includes A or B" is satisfied.Furthermore, the use of the article "a," a," or "an," in this application and the appended claims generally should be interpreted to mean "one or more," unless otherwise specified or clear from the context. Furthermore, the consistent use of the term "an implementation" or "the implementation" is not intended to mean the same as embodiment or implementation unless specifically identified as such.

[0057] Implementations of transmitting station 202 and / or receiving station 206 (and the algorithms, methods, instructions, etc. stored thereon and / or executed thereby, including by encoding device 500 and decoding device 600) may be realized using hardware, software, or a combination thereof. The hardware may include, for example, computers, intellectual property (IP) cores, application-specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, and any other suitable circuitry. In the claims, the term "processor" should be understood to encompass any of the aforementioned hardware, whether individually or in combination. The terms "signal" and "data" are used interchangeably.Furthermore, the sections of the transmitting station 202 and the receiving station 206 do not necessarily have to be implemented in the same manner.

[0058] Furthermore, for example, in one aspect, transmitting station 202 or receiving station 206 may be implemented using a general-purpose computer or a general-purpose processor with a computer program that, when executed, performs any corresponding methods, algorithms, and / or applications described herein. Additionally or alternatively, for example, a dedicated computer / processor may be used, which may include other hardware, to perform any of the methods, algorithms, or instructions described herein.

[0059] The transmitting station 202 and the receiving station 206 may be implemented, for example, on computers in a video conferencing system. Alternatively, the transmitting station 202 may be implemented on a server, and the receiving station 206 may be implemented on a device separate from the server, such as a handheld communication device. In this case, the transmitting station 202 may encode content into an encoded video signal using an encoding device 500 and transmit the encoded video signal to the communication device. In turn, the communication device may then decode the encoded video signal using a decoding device 600. Alternatively, the communication device may decode content stored locally on the communication device, such as content that was not transmitted by the transmitting station 202.Other suitable transmitting or receiving implementation schemes are available. For example, receiving station 206 may generally be a stationary personal computer rather than a portable communications device, and / or a device including an encoding device 500 may also include a decoding device 600.

[0060] Furthermore, all or a portion of the implementations of the present invention may take the form of a computer program product, accessible, for example, from a tangible computer-usable or computer-readable medium. A computer-usable or computer-readable medium may be any device that can contain, store, communicate, or transport the program, for example, in a tangible manner for use by or in connection with any processor. The medium may be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable media are also available.

[0061] According to embodiments, dynamic resolution switching achieves a target bit rate for single-pass and two-pass encoding of a video stream. Single-pass encoding determines whether an encoding bit rate for a frame meets a target bit rate. If not, a quantification parameter used to encode the frame is compared to minimum and maximum thresholds to determine whether the video's resolution needs to be adjusted. Two-pass encoding encodes an input frame using a quantification parameter and determines whether the video's resolution needs to be adjusted for encoding the frame during a second pass based on the bit rate at which the input frame is encoded. The resolution may be adjusted based on the encoding device constraints with reference to a motion search range used to encode a frame.

[0062] The above-described embodiments, implementations, and aspects have been described to facilitate a simple understanding of the present invention and do not limit the present invention. On the contrary, the invention is intended to cover various modifications and equivalent arrangements within the scope of the appended claims, the scope of which is to be given the broadest interpretation to encompass all such modifications and equivalent arrangements as permitted by law.

Claims

[1] A method (700) for coding rate control of real-time videos, comprising: identifying a first frame of a plurality of frames of a video data stream; encoding (704) the first frame at a first bit rate (108) representative of a quantification parameter (106); comparing (710) the first bit rate (108) with a target bit rate (104) for the first frame and comparing (712; 718) the quantification parameter (106) with either a minimum quantification threshold (100) or a maximum quantification threshold (102), wherein if the first bit rate (108) is greater than the target bit rate (104), the quantification parameter (106) is compared with the maximum quantification threshold (102) and if the first bit rate (108) is less than the target bit rate (104), the quantification parameter (106) is compared with the minimum quantification threshold (100); adjusting (714; 716; 720; 722) at least one coding parameter in response to comparing (710) the first bit rate (108) with the target bit rate (104) and comparing (712; 718) the quantification parameter (106) with either the minimum quantification threshold (100) or the maximum quantification threshold (102); identifying a second frame of a plurality of frames of the video data stream; encoding the second frame at a second bit rate using the adjusted at least one encoding parameter; and transmitting the encoded first frame and the encoded second frame in an output bitstream. [2] The method (700) of claim 1, wherein, if the first bit rate (108) is less than the target bit rate (104) and the quantification parameter (106) is greater than the minimum quantification threshold (100), adjusting (720) the at least one coding parameter comprises: lowering the quantification parameter (106). [3] The method (700) according to claim 1 or 2, wherein, if the first bit rate (108) is lower than the target bit rate (104) and the quantification parameter (106) is at most equal to the minimum quantification threshold (100), adjusting (722) the at least one coding parameter comprises: increasing the resolution of the video stream; increasing the quantification parameter (106); and adjusting the minimum quantification threshold (100) and the maximum quantification threshold (102) based on the increased resolution. [4] The method (700) of any one of claims 1 to 3, wherein, if the first bit rate (108) is greater than the target bit rate (104) and the quantification parameter (106) is less than the minimum quantification threshold (100), adjusting (714) the at least one coding parameter comprises: increasing the quantification parameter (106). [5] The method (700) of any one of claims 1 to 4, wherein, if the first bit rate (108) is greater than the target bit rate (104) and the quantification parameter (106) is at least as large as the maximum quantification threshold (102), adjusting (716) the at least one coding parameter comprises: reducing the resolution of the video stream; lowering the quantification parameter (106); and adjusting the minimum quantification threshold (100) and the maximum quantification threshold (102) based on the reduced resolution. [6] Method (700) according to any one of claims 1 to 5 comprising: identifying one or more coding constraints with respect to a motion search region in response to coding the first frame, wherein adjusting the at least one coding parameter comprises: reducing a resolution of the video data stream based on the one or more encoding device limitations; determining a reference location of the first frame at the reduced resolution; and re-setting a center location of the motion search area based on the reference location of the first frame. [7] The method (700) of any one of claims 1 to 6, wherein the at least one coding parameter comprises at least one of the following values: a resolution of the video data stream, the quantification parameter (106), the minimum quantification threshold (100), or the maximum quantification threshold (102). [8] Apparatus (500) for carrying out the method (700) according to any one of the preceding claims. [9] The apparatus (500) of claim 8 comprising at least one processor (302) configured to execute instructions that cause the processor (302) to execute the method (700) of any one of claims 1 to 7, wherein the instructions are stored on a non-tangible storage medium (304).

Images