A ranked reference framework for video coding
The ranked reference framework addresses inefficiencies in managing reference frames by evaluating their usefulness for video encoding and decoding, leading to improved coding efficiency through accurate frame selection and simplified entropy coding.
Patent Information
- Application Number
- JP2024534081
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-07
- Filing Date
- 2022-12-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-12-07
AI Technical Summary
Existing video encoding and decoding technologies face inefficiencies in managing reference frames, leading to suboptimal coding efficiency due to reliance on label-based identification of reference frames that may not accurately reflect their actual usefulness for prediction, complicating signaling and reducing coding performance.
A ranked reference framework that evaluates the usefulness of each possible reference frame based on heuristics such as temporal distance, quality level, and selection frequency, ranking frames for efficient inter prediction without relying on named reference mappings, simplifying context trees and entropy coding.
This approach enhances coding efficiency by accurately selecting the most suitable reference frames for prediction, reducing complexity in signaling and improving entropy coding performance.
Smart Images

Figure 2024542839000001_ABST
Abstract
Description
[Background technology]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 286,587, filed December 7, 2021, the disclosure of which is incorporated by reference in its entirety herein.
[0002] A digital video stream may display video using a series of frames or still images. Digital video can be used for a variety of applications, such as video conferencing, high-definition video entertainment, video advertising, or sharing user-generated video. Digital video streams can contain large amounts of data and consume large amounts of computing or communication resources of a computing device to process, transmit, or store the video data. Various approaches have been proposed to reduce the amount of data in video streams, such as compression and other coding techniques. These techniques can include both lossy and lossless coding techniques. Summary of the Invention
[0003] The present disclosure relates generally to encoding and decoding video data using reference frames, and more specifically to encoding and decoding blocks of video frames using a ranked reference framework of reference frames.
[0004] An aspect of the teachings herein is a method of coding (i.e., encoding or decoding) an identifier of at least one reference frame used to code a current block of a video frame. The method can include identifying a syntax element to be coded, the syntax element corresponding to the identifier, determining context information for the syntax element, the context information including available reference frames used to code a block preceding the current block in coding order, determining a node of a context tree that includes the syntax element, and coding the syntax element according to a probabilistic model using the context information associated with the node. The context tree is a binary tree that includes available reference frames as nodes arranged in a ranking based on a score calculated for each reference frame using at least one characteristic of the reference frame that indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames.
[0005] In some embodiments, the at least one characteristic comprises at least two different characteristics.
[0006] In some implementations, the method includes determining whether a coding mode of the current block is a single reference coding mode or a mixed reference coding mode, and the context tree has the same syntactic structure regardless of whether the coding mode of the current block is a single reference coding mode or a mixed reference coding mode.
[0007] In some implementations, the available reference frames included in the context tree include forward and backward reference frames.
[0008] In some implementations, the context tree is a single context tree including a forward reference frame and a backward reference frame, the coding mode of the current block is a mixed reference coding mode, the identifier includes a first identifier of a first reference frame and a second identifier of a second reference frame, identifying a syntax element to be coded includes identifying a first syntax element corresponding to the first identifier and identifying a second syntax element corresponding to the second identifier, determining a node includes determining a first node of the single context tree including the first syntax element and determining a second node of the single context tree including the second syntax element, determining context information for the syntax element includes determining first context information for the first syntax element and determining second context information for the second syntax element, and coding the syntax element includes coding the first syntax element according to a first probability model using the first context information and coding the second syntax element according to a second probability model using the first context information.
[0009] In some implementations, the at least one characteristic includes a quality level of the reference frame of the available reference frames being ranked and a temporal distance of the reference frame from the video frame containing the current block.
[0010] In some implementations, the frame header of the video frame that contains the current block includes a mask having a bit length equal to the cardinality of the available reference frames, which indicates which of the reference frames available to the current block may be used for inter prediction and which of the reference frames available to the current block may not be used for inter prediction.
[0011] Another aspect of the teachings herein is a method of decoding a current block of a video frame. The method can include ranking a list of available reference frames according to at least one characteristic of each reference frame, decoding an identifier of at least one reference frame used to encode the current block from a block header of the current block using a context tree, and decoding the current block using the at least one reference frame. The context tree is a single binary tree with nodes arranged from the highest ranked reference frame of the available reference frames to the lowest ranked reference frame of the available reference frames.
[0012] In some implementations, the at least one characteristic of each reference frame indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames.
[0013] In some implementations, the at least one characteristic includes at least two of: a temporal distance from a video frame containing the current block, a quality or quantization level of the reference frame, a number of times the reference frame has been selected for inter prediction, or gradient information of the reference frame.
[0014] Yet another aspect of the teachings herein is a method of encoding a current block of a video frame. The method can include ranking a list of available reference frames based on a score calculated for each reference frame using at least one characteristic of the reference frame that indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames, encoding the current block using at least one of the available reference frames, and encoding an identifier of the at least one reference frame used to encode the current block using a context tree into a block header of the current block. The context tree is a single binary tree with nodes arranged from the highest ranked reference frame of the available reference frames to the lowest ranked reference frame of the available reference frames.
[0015] In some implementations, encoding the current block includes encoding the current block multiple times using a single reference coding mode and each of the available reference frames except for at least the lowest ranked reference frame, and selecting at least one of the reference frames as a single reference frame of the available reference frames having the highest coding efficiency.
[0016] In some implementations, encoding the current block using at least one reference frame includes encoding the current block using two reference frames, and encoding the identifier includes encoding a first identifier for a first reference frame of the two reference frames using a single binary tree and encoding a second identifier for a second reference frame of the two reference frames using a single binary tree.
[0017] In some implementations, the available reference frames of the single binary tree include a forward reference frame and a backward reference frame.
[0018] Apparatus capable of carrying out any of these methods is also described.
[0019] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims, and the accompanying drawings.
[0020] This description makes reference to the accompanying drawings, which are listed below, in which like reference numerals refer to like parts throughout the several views. [Brief description of the drawings]
[0021] [Figure 1] 1 is a schematic diagram of a video encoding and decoding system. [Diagram 2] 1 is a block diagram illustrating an example of a computing device in which a sending station or a receiving station can be implemented. [Diagram 3] FIG. 2 shows an example of a video stream that is encoded and then decoded. [Figure 4] FIG. 2 is a block diagram of an encoder according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a block diagram of a decoder according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating an example of a reference frame buffer. [Figure 7] FIG. 1 shows an example of a one-layer coding structure. [Figure 8] FIG. 13 illustrates an example of a context tree for entropy coding a reference frame identifier for a block encoded using a single reference prediction mode. [Figure 9] FIG. 13 illustrates an example of a context tree for entropy coding a frame identifier for a block encoded using a unidirectional mixed reference prediction mode. [Figure 10A] FIG. 13 illustrates an example of a context tree for entropy coding a forward frame identifier for a block encoded using a bidirectional mixed reference prediction mode. [Figure 10B] FIG. 13 illustrates an example of a context tree for entropy coding a backward frame identifier for a block encoded using a bidirectional mixed reference prediction mode. [Figure 11] 1 is a flow chart diagram of a method for coding one or more reference frames of an inter-predicted block. [Figure 12] FIG. 2 is a diagram illustrating an example of a context tree for entropy coding one or more reference frames of an inter-predicted block. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] Compression techniques associated with coding video content (e.g., video streams, video files, etc.) may involve dividing each image into blocks and generating a digital video output bitstream using one or more techniques to limit the information included in the output. The received bitstream may be decoded to recreate the blocks and source image from the limited information. When encoding a video stream, or a portion thereof (e.g., a frame or block), temporal and spatial similarities within the video stream may be used to improve coding efficiency. For example, temporal similarities may be exploited when encoding a current block of a current frame by predicting motion and color information of the current block based on previously encoded pixels of one or more other frames and identifying the difference between the predicted value (prediction block) and the current block (residual). In this method, only the residual and parameters used to generate the prediction block need to be added to the bitstream, rather than including the entire current block. This technique is sometimes referred to as inter-prediction.
[0023] This process of generating predictive blocks of pixels for a current frame from other frames in a video sequence involves maintaining a set of reference frames. The reference buffer management system or scheme used to maintain the set may involve many decisions, such as determining which frames to keep as references, when to discard references in favor of new references, how many references are available for the current frame or prediction unit (e.g., a block), how to signal which references are used for a prediction unit, and how to efficiently search for relevant references in a buffer pool for prediction. The design of this scheme is useful for proper block prediction (e.g., a predicted block that closely matches the current block being encoded), and the high degree of flexibility of the scheme allows it to make the most of what is allowed by the bitstream specification of a particular codec.
[0024] There are several known reference buffer management systems or schemes (also called reference picture management schemes), each associated with a codec. For example, H.265 / HEVC orders the reference picture list based on the display order of the reference frames. This ordering provides a relatively simple scheme for maintaining, updating, and signaling the reference frames. However, using only distance to order the reference picture list can result in some coding inefficiencies in the signaling of the reference frames.
[0025] In contrast, in the VP9 reference buffer management scheme, a prediction unit can choose from three possible reference frames that can be dynamically selected from a pool of eight frame buffers. These three possible reference frames are named (e.g., labeled or otherwise identified) LAST, GOLDEN, and ALTREF, respectively. AV1 also uses dynamic referencing in its reference buffer management scheme that selects a reference frame from a pool of eight frame buffers. However, AV1 allows seven prediction unit references. These seven possible reference frames are named (e.g., labeled or otherwise identified) LAST, LAST2, LAST3, GOLDEN, BWDREF (or ALTREF3), ALTREF2, and ALTREF, respectively. These are sometimes referred to herein as reference frame names or labels. While the dynamic reference buffer management scheme in each of the latter two codecs allows flexibility in the selection of reference frames for efficient coding, the identification of a buffer by a label implies a particular quality or statistic regarding the associated reference frame, as explained in more detail below. These qualities or statistics may not accurately reflect the actual reference frames associated with each reference frame label. Furthermore, label identification complicates reference frame signaling.
[0026] This disclosure describes an embodiment of a reference frame management scheme that relies on a ranked list of n possible references. The ranking can be performed using a heuristic designed to determine the usefulness of each possible reference frame, its likelihood of being a good choice for predicting a block in the current frame (e.g., its likelihood of minimizing the residual of the block), or a combination thereof. The heuristic considers multiple frame statistics, including or instead of the distance of the reference frame from the current frame, and therefore may provide improved coding efficiency over existing approaches. Furthermore, the reference frame management scheme is simplified and more flexible than schemes that use label-based identification of reference frames.
[0027] Further details of the reference frame management scheme are described herein first with reference to a system in which it may be implemented.
[0028] Figure 1 is a schematic diagram of a video encoding and decoding system 100. The sending station 102 can be, for example, a computer having a hardware internal configuration as described in Figure 2. However, other suitable implementations of the sending station 102 are possible. For example, the processing of the sending station 102 can be distributed across multiple devices.
[0029] A network 104 may connect the sending station 102 and the receiving station 106 for encoding and decoding the video stream. Specifically, the video stream is encoded at the sending station 102, and the encoded video stream is decoded at the receiving station 106. The network 104 may be, for example, the Internet. The network 104 may also be a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), a cellular network, or other means of transferring the video stream from the sending station 102 to the receiving station 106 in this example.
[0030] Receiving station 106 may, in one example, be a computer with hardware internals as described in Figure 2. However, other suitable implementations of receiving station 106 are possible. For example, the processing of receiving station 106 may be distributed across multiple devices.
[0031] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation may omit the network 104. In another implementation, the video stream may be encoded and stored for later transmission to the receiving station 106 or other device with memory. In one implementation, the receiving station 106 receives the encoded video stream (e.g., via the network 104, a computer bus, and / or some other communication path) and stores the video stream for later decoding. In one embodiment, the Real-time Transport Protocol (RTP) is used to transmit the encoded video over the network 104. In another implementation, a transport protocol other than RTP (e.g., the HyperText Transfer Protocol (HTTP) video streaming protocol) may be used.
[0032] For example, when used in a videoconferencing system, the sending station 102 and / or the receiving station 106 may include the functionality to encode and decode video streams as described below. For example, the receiving station 106 may be a videoconferencing participant that receives, decodes, and displays an encoded video bitstream from a videoconferencing server (e.g., the sending station 102), and also encodes and sends its own video bitstream to the videoconferencing server for decoding and display by other participants.
[0033] Figure 2 is a block diagram illustrating an example of a computing device 200 (e.g., an apparatus) that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of Figure 1. The computing device 200 can be in the form of a computing system that includes multiple computing devices, or in the form of a single computing device, such as, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.
[0034] CPU 202 in computing device 200 may be a conventional central processing unit, or CPU 202 may be any other type of device or devices now existing or later developed that can manipulate or process information. Although the disclosed embodiments may be implemented with a single processor, such as CPU 202 as illustrated, advantages of speed and efficiency may be achieved using multiple processors.
[0035] The memory 204 in the computing device 200 may be a read-only memory (ROM) device or a random access memory (RAM) device in an embodiment. Any other suitable type of storage device may be used as the memory 204. The memory 204 may include code and data 206 accessed by the CPU 202 using the bus 212. The memory 204 further includes an operating system 208 and an application program 210, which may include at least one program that enables the CPU 202 to execute the methods described herein. For example, the application program 210 may include applications 1 to N and may further include a video coding application that executes the methods described herein. The computing device 200 also includes a secondary storage 214, which may be, for example, a memory card used in a mobile computing device. Since a video communication session may include a large amount of information, all or part of it may be stored in the secondary storage 214 and loaded into the memory 204 as needed for processing.
[0036] Computing device 200 may also include one or more output devices, such as a display 218. Display 218, in one example, may be a touch-sensitive display that combines a display with touch-sensing elements operable to sense touch input. Display 218 may be connected to CPU 202 via bus 212. In addition to or instead of display 218, other output devices may be provided that enable a user to program or otherwise use computing device 200. When the output device is or includes a display, the display may be implemented in a variety of ways, such as a liquid crystal display (LCD), a cathode ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.
[0037] Computing device 200 may also include or communicate with image sensing device 220, such as a camera or any other image sensing device now existing or later developed, that can sense images, such as an image of a user operating computing device 200. Image sensing device 220 may be positioned to be pointed at a user operating computing device 200. In one example, the position and optical axis of image sensing device 220 may be configured such that the field of view is immediately adjacent to display 218 and includes the area from which display 218 is viewable.
[0038] Computing device 200 may include or communicate with sound sensing device 222, such as a microphone, or any other now existing or later developed sound sensing device that can detect sounds proximate computing device 200. Sound sensing device 222 may be positioned to face a user operating computing device 200 and configured to receive sounds, e.g., voice or other speech, made by the user while the user is operating computing device 200.
[0039] Although in FIG. 2 the CPU 202 and memory 204 of computing device 200 are shown as integrated into one unit, other configurations are available. The operations of CPU 202 may be distributed across multiple machines (each machine may have one or more processors) that may be coupled directly or through a local area network or other network. Memory 204 may be distributed across multiple machines, such as a network-based memory or memory in multiple machines that perform the operations of computing device 200. Although shown here as one bus, bus 212 of computing device 200 may be comprised of multiple buses. Additionally, secondary storage 214 may be directly connected to other components of computing device 200 or may be accessed over a network and may include an integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, computing device 200 may be implemented in a variety of configurations.
[0040] FIG. 3 illustrates an example of a video stream 300 that is encoded and then decoded. The video stream 300 includes a video sequence 302. At a next level, the video sequence 302 includes a number of adjacent frames 304. Although three frames are shown as adjacent frames 304, the video sequence 302 may include any number of adjacent frames 304. The adjacent frames 304 may be further subdivided into individual frames, for example frame 306. At a next level, the frame 306 may be divided into a series of planes or segments 308. The segments 308 may be subsets of frames that allow for parallel processing, for example. The segments 308 may be subsets of frames that allow for separation of the video data into individual colors. For example, a frame 306 of color video data includes a luma plane and two chroma planes. The segments 308 may be sampled at different resolutions.
[0041] Regardless of whether a frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which may contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 may also be arranged to contain data from one or more segments 308 of pixel data. The blocks 310 may also be of any other suitable size, such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger sizes. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.
[0042] 4 is a block diagram of an encoder 400 according to an embodiment of the present disclosure. The encoder 400 may be implemented in the sending station 102, such as by providing a computer software program stored in a memory, such as memory 204, as described above. The computer software program may include machine instructions that, when executed by a processor, such as CPU 202, cause the sending station 102 to encode video data in the manner described in FIG. 4. The encoder 400 may also be implemented as dedicated hardware, such as included in the sending station 102. In one particularly preferred embodiment, the encoder 400 is a hardware encoder.
[0043] The encoder 400 has the following stages to perform various functions in a forward path (shown by solid connecting lines): intra / inter prediction stage 402, transform stage 404, quantization stage 406, and entropy encoding stage 408, to generate an encoded or compressed bitstream 420 using the video stream 300 as input. The encoder 400 may also include a reconstruction path (shown by dotted connecting lines) to reconstruct frames for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform various functions in the reconstruction path: inverse quantization stage 410, inverse transform stage 412, reconstruction stage 414, and loop filtering stage 416. Other structural variations of the encoder 400 may also be used to encode the video stream 300.
[0044] When the video stream 300 is presented for encoding, each frame 304, such as frame 306, can be processed block by block. In the intra / inter prediction stage 402, each block can be encoded using intraframe prediction (also called intraprediction) or interframe prediction (also called interprediction). In either case, a predictive block can be formed. In the case of intraprediction, the predictive block may be formed from previously encoded and reconstructed samples of the current frame. In the case of interprediction, the predictive block may be formed from samples in one or more previously constructed reference frames.
[0045] Next, still referring to FIG. 4, the prediction block may be subtracted from the current block in an intra / inter prediction stage 402 to generate a residual block (also referred to as a residual). A transform stage 404 converts the residual into transform coefficients, e.g., in the frequency domain, using a block-based transform. A quantization stage 406 converts the transform coefficients into discrete quantum values, called quantized transform coefficients, using a quantization value or quantization level. For example, the transform coefficients may be divided by the quantization value and truncated. The quantized transform coefficients are entropy encoded by an entropy encoding stage 408. The entropy encoded coefficients are output to a compressed bitstream 420 along with other information used to decode the block (which may include, e.g., the type of prediction used, the transform type, the motion vectors, and the quantization value). The compressed bitstream 420 may be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 may also be referred to as an encoded video stream or encoded video bitstream, and these terms are used interchangeably herein.
[0046] The reconstruction path in FIG. 4 (represented by the dotted connecting lines) may enable the encoder 400 and the decoder 500 (described below) to use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions similar to those performed during the decoding process described in detail below, including inverse quantization of the quantized transform coefficients in the inverse quantization stage 410 and inverse transforming the inverse quantized transform coefficients in the inverse transform stage 412 to generate a differential residual block (also called a differential residual). In the reconstruction stage 414, a prediction block predicted in the intra / inter prediction stage 402 may be added to the differential residual to create a reconstructed block. A loop filtering stage 416 may be applied to the reconstructed block to reduce distortions such as blocking artifacts.
[0047] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. For example, a non-transform based encoder can directly quantize the residual signal for a particular block or frame without using the transform stage 404. In another implementation, the encoder may have the quantization stage 406 and the inverse quantization stage 410 combined into a common stage.
[0048] Figure 5 is a block diagram of a decoder 500 according to an embodiment of the present disclosure. The decoder 500 may be implemented in the receiving station 106, for example, by providing a computer software program stored in memory 204. The computer software program may include machine instructions that, when executed by a processor such as CPU 202, cause the receiving station 106 to decode video data in the manner described in Figure 5. The decoder 500 may also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.
[0049] The decoder 500, similar to the reconstruction path of the encoder 400 described above, in one example includes the following stages that perform various functions to generate an output video stream 516 from a compressed bitstream 420: an entropy decoding stage 502, an inverse quantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a deblocking filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.
[0050] When the compressed bitstream 420 is presented for decoding, data elements in the compressed bitstream 420 may be decoded by an entropy decoding stage 502 to generate a set of quantized transform coefficients. An inverse quantization stage 504 inverse quantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by a quantization value), and an inverse transform stage 506 inverse transforms the inverse quantized transform coefficients to generate differential residuals that may be identical to those created by the inverse transform stage 412 of the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 may use an intra / inter prediction stage 508 to create a prediction block that is identical to that created by the encoder 400 (e.g., the intra / inter prediction stage 402). In a reconstruction stage 510, a prediction block may be added to the differential residuals to create a reconstructed block. A loop filtering stage 512 may be applied to the reconstructed block to reduce blocking artifacts.
[0051] Other filtering may be applied to the reconstructed blocks. In this example, a deblocking filtering stage 514 is applied to the reconstructed blocks to reduce blocking artifacts and the result is output as an output video stream 516. The output video stream 516 may also be referred to as a decoded video stream, and these terms are used interchangeably herein. Other variations of the decoder 500 may be used to decode the compressed bitstream 420. For example, the decoder 500 may generate the output video stream 516 without the deblocking filtering stage 514.
[0052] 6 is a block diagram illustrating an example of a reference frame buffer 600 that can hold up to eight buffers. Reference frame buffer 600 stores reference frames that are used to encode or decode blocks of frames of a video sequence. In this example, labels, roles, or types may be associated with or used to describe the various reference frames identified (e.g., associated with, stored, etc.) in each buffer of reference frame buffer 600.
[0053] The frame header of the current reference frame may include a virtual index 608 to the location in the reference frame buffer 600 where the reference frame is stored. A reference frame mapping 612 may map the virtual index 608 of the reference frame to a physical index 614 of the memory where the reference frame is stored. One or more refresh flags 610 may be used to remove one or more stored reference frames from the reference frame buffer 600, for example, to clear space in the reference frame buffer 600 for a new reference frame when there are no more blocks to encode or decode using the stored reference frames, when a new key frame is to be encoded or decoded, or a combination thereof. In an example of a frame-level reference syntax, a flag (e.g., 8 bits) indicates which buffers in the reference frame buffer 600 to update with the current frame. Another component of the frame-level reference syntax is a list of values indicating the mapping of buffers to named reference allocations. In this example, the reference frames are allocated, named, identified, or designated as the LAST frame 602, the GOLDEN frame 604, and the alternate reference (ALTREF) frame 606.
[0054] In general, a group of frames of a video sequence having a display order may be coded starting with a key frame. A key frame is a frame that is coded without reference to other frames (e.g., its blocks are coded using intra prediction). When the term coding or variations thereof are used herein, the term refers to either encoding or decoding unless the context clearly indicates otherwise. Once reconstructed, a key frame can be used as a reference frame for coding blocks of one or more subsequent frames of the coded group in a coding order that may differ from the display order. FIG. 7 illustrates an example of a one-layer coding structure 700 that can be used to explain how a group of frames is coded. In FIG. 7, more than three named references may be dynamically selected from a pool of eight buffers, such as reference frame buffer 600.
[0055] An encoder, such as the encoder 400 of FIG. 4, may encode a group of frames according to the one-layer coding structure 700. A decoder, such as the decoder 500 of FIG. 5, may decode the group of frames using the one-layer coding structure 700. The decoder may receive an encoded bitstream, such as the compressed bitstream 420 of FIG. 5. In the encoded bitstream, the frames of the group of frames may be ordered (e.g., arranged, stored, etc.) in the coding order of the one-layer coding structure 700. The decoder may decode the frames in the one-layer coding structure 700 and display them according to a display order. The encoded bitstream may include syntax elements that the decoder can use to determine the display order.
[0056] The group of frames in FIG. 7 is shown in display order as frame 702, frame 704, frame 706, frame 708, frame 710, frame 712, frame 714, frame 716, frame 718, and frame 720. The numbers in the boxes indicate the coding order of the frames. As mentioned above, a group of frames of a video sequence for coding typically starts with a key frame. Once reconstructed, the key frame can be used as a reference frame for coding blocks of one or more subsequent frames of the group coded in coding order. An indication (e.g., in the frame's header) that a frame is a key frame may issue a command to update all frames for coding, such as updating all frames in the reference frame buffer 600. There are up to five named reference frames (GOLDEN, ALTREF, LAST, LAST2, and LAST3) for inter prediction, which are described in more detail below.
[0057] In this example, frame 702 is a key frame. The reconstructed key frame 702 may be identified as a GOLDEN frame. For example, the reconstructed key frame 702 may be assigned to a buffer using an 8-bit bitmap indicating which of the 8 buffers to update with the current frame (e.g., reconstructed key frame 702), and its 3-bit index may be used to map the buffer to a named reference slot GOLDEN. The GOLDEN frame may be used to code a subsequent frame in the group. For example, the next frame in the coding order may be a frame that is a defined number of frames ahead of the key frame 702 in the display order, e.g., a future frame. In FIG. 7, the second frame in the coding order is frame 720, which is the last frame in the frame group. Blocks of frame 720 may use the GOLDEN frame for inter prediction. The reconstructed frame 720 may be identified as an ALTREF frame and may be temporally filtered.
[0058] The third frame in the coding order of FIG. 7 is frame 704, which is the second frame in display order. The blocks of frame 704 can use the GOLDEN frame, the ALTREF frame, or both for inter prediction. The reconstructed frame 704 may be identified as the LAST frame for coding frame 706, which is the next frame in display order and also the next frame in coding order. Frame 706 has the GOLDEN frame, the ALTREF frame, and the LAST frame available for inter prediction. If there are more than three named reference slots, the reconstructed frame 706 may be identified as a reference frame. In this example, the frame header reference syntax may refresh the buffer such that the reconstructed frame 706 is identified as the LAST frame for coding frame 708, which is the next frame in display order and also the next frame in coding order, and the reconstructed frame 704 is identified as the LAST2 frame. Frame 708 has the GOLDEN frame, the ALTREF frame, the LAST frame, and the LAST2 frame available for inter prediction. When coding frame 710, the frame header reference syntax can update the buffers such that reconstructed frame 708 is identified as the LAST frame, reconstructed frame 706 is identified as the LAST2 frame, and reconstructed frame 704 is identified as the LAST3 frame.
[0059] This process of updating buffers and coding frames continues. For example, as indicated by the arrows in Figure 7, frame 702 identified as the GOLDEN frame, frame 720 identified as the ALTREF frame, frame 714 identified as the LAST frame, frame 712 identified as the LAST2 frame, and frame 710 identified as the LAST3 frame can be used to code the block of frame 716. Once the last frame in the coding order has been coded, coding is complete.
[0060] 7 is an example of a frame 720 in which only one future frame for backward prediction is identified, namely, the ALTREF frame. Other coding structures can be used to code a group of frames using additional reference frames, including additional future frames (e.g., relative to the current frame), such as, for example, ALTREF2 and BWDREF. In some implementations, a multi-layer coding structure may be used.
[0061] A prediction unit of a frame, referred to as a current block in these examples, can be predicted in the intra / inter prediction stage 402 using inter prediction as described above with respect to the example of FIG. 4. Inter prediction can include a single reference frame prediction mode or a mixed reference frame prediction mode. In some implementations, up to seven reference frames may be available when coding a block using a single reference frame prediction mode or a mixed reference frame prediction mode. For a mixed reference frame prediction mode, a combination of reference frames with respective motion vectors may be used. In one example, any two reference frames may be used in a mixed reference frame prediction mode. Thus, any combination of two of the seven available reference frames (e.g., 28 possible combinations) can be used. In another example, only a subset of all possible combinations (e.g., used for coding the current block) may be valid.
[0062] When encoding a current block into a bitstream, such as bitstream 420, parameters and / or identifiers of parameters (e.g., syntax elements) used to encode the current block, such as motion vectors and reference frames, are entropy encoded, such as in entropy encoding stage 408. A common approach to entropy coding considers the context (e.g., associated values) of the current block, and may also consider the context of neighboring blocks. Here, we use context-adaptive binary arithmetic coding (CABAC) to illustrate examples of the present teachings. However, the present teachings are not so limited and can be applied to other entropy coding schemes that use context, such as, for example, context-adaptive variable length coding (CAVLC) and those that use non-binary context trees.
[0063] In general, CABAC encodes these block-level syntax elements as binary symbols, allowing probability modeling of the more frequently used bits of any symbol. The probability model is adaptively selected based on the local context (e.g., nearby elements), allowing for better modeling of the probability estimates. The local context of the probability modeling of syntax elements corresponding to reference frames relates to, for example, whether and how many neighboring blocks are coded using a single reference frame prediction mode compared to a mixed reference frame prediction mode, as well as how many and which reference frames are used to code the neighboring blocks.
[0064] Named reference buffers or slots are intended to indicate certain qualities or statistics about the associated reference frame. For example, LAST, LAST2, LAST3, and GOLDEN are considered to be past reference frames of increasing distance from the current frame in the display order of a group of frames or pictures in a video sequence, while BWDREF, ALTREF2, and ALTREF are considered to be future reference frames of increasing display order from the current frame. As explained at the beginning, reference buffer management schemes using named slots or buffers can lead to coding inefficiencies. One reason for this is that there is no guarantee that the reference frame identified by a particular named slot has the properties that its name is intended to indicate. This can be accounted for by coding reference frames as block-level syntax elements.
[0065] First, there is a different syntax in which a current block is coded using a mixed reference prediction mode compared to a single reference prediction mode. The bitstream may signal whether the mode is a single reference prediction mode or a mixed reference prediction mode, for example by a flag in the block header. FIG. 8 illustrates an example of a context tree for entropy coding a reference frame identifier for a block encoded using a single reference prediction mode. The context tree 800 describes the binary code for each possible reference frame slot. Each node P1-P6 of the context tree is an entropy coded bit with a different probability model (e.g., cumulative distribution function) and context. That is, each node P1-P6 represents a different way of computing the context for a particular binary decision at node P1-P6.
[0066] The context may be determined using information of blocks neighboring the current block. The neighboring blocks may be blocks that are physically or temporally adjacent to the current block and coded before the current block. In some implementations, the neighboring blocks may be one or more blocks to the left of the current block located in the current frame, one or more blocks above the current block located in the current frame, one or more blocks in a previous frame that are co-located with the current block, or a combination thereof.
[0067] First, the cardinality of the forward reference frame used to code the adjacent block (i.e., in this example, LAST, LAST2, LAST3, GOLDEN) and the cardinality of the backward reference frame used to code the adjacent block (i.e., in this example, BWDREF, ALTREF2, ALTREF) may be compared at node P1. The cardinality of the forward reference frame used to code the adjacent block may be determined by summing each instance where the adjacent frame uses any one of LAST, LAST2, LAST3, or GOLDEN. The cardinality of the backward reference frame used to code the adjacent block may be determined by summing each instance where the adjacent frame uses any one of BWDREF, ALTREF2, or ALTREF. If the cardinality of the forward reference frame is less than the cardinality of the backward reference frame, the context value may be set to 0, if the cardinality of the forward reference frame is equal to the cardinality of the backward reference frame, the context value may be set to 1, and if the cardinality of the forward reference frame is greater than the cardinality of the backward reference frame, the context value may be set to 2.
[0068] The cardinality of the BWDREF and ALTREF2 reference frames used for coding the neighboring blocks and the cardinality of the ALTREF reference frame used for coding the neighboring blocks may be compared at node P2. If the cardinality of the BWDREF and ALTREF2 reference frames is less than the cardinality of the ALTREF reference frame, the context value may be set to 0, if the cardinality of the BWDREF and ALTREF2 reference frames is equal to the cardinality of the ALTREF reference frame, the context value may be set to 1, and if the cardinality of the BWDREF and ALTREF2 reference frames is greater than the cardinality of the ALTREF reference frame, the context value may be set to 2. The remaining nodes of the context tree 800 may be considered in a similar manner.
[0069] In mixed reference prediction mode, different syntax may be used when the current block is predicted bidirectionally (i.e., a combination of past and future reference frames are used for backward and forward prediction) compared to when the current block is predicted unidirectionally (i.e., either past or future reference frames are used for prediction, but not both). FIG. 9 illustrates an example of a context tree 900 for entropy coding frame identifiers for blocks encoded using unidirectional mixed reference prediction mode. The context tree 900 describes binary codes for combinations of reference frame slots. Each node P0-P2 of the context tree is an entropy coded bit with a different probability model (e.g., cumulative distribution function) and context. In other words, each node P0-P2 represents a different computation method of the context for a particular binary decision at node P0-P2.
[0070] Context is calculated using the context tree 900 using a strategy similar to that described with respect to the context tree 800. That is, the context value of node P0 may be based on a comparison of the cardinality of the forward reference frame used to code the neighboring block and the cardinality of the backward reference frame used to code the neighboring block. In some implementations, the count may be limited to neighboring blocks predicted using a mixed reference prediction mode. To reduce coding complexity, the complete set of named frames may not be used. For example, as shown in the example of FIG. 9, in a unidirectional mixed reference prediction mode, ALTREF2 may not be considered. In some implementations, to reduce coding complexity, the combination of forward (e.g., past) reference frames, the combination of backward (e.g., future) reference frames, or both may be limited. For example, as shown in the example of FIG. 9, each of the combinations of two past reference frames used in the unidirectional mixed reference prediction mode includes LAST. Thus, the combinations of LAST2, LAST3, and GOLDEN are not included in the context tree 900.
[0071] In contrast to the context tree 800 for the single reference prediction mode shown in the example of Figure 8 and the context tree 900 for the unidirectional mixed reference prediction mode shown in the example of Figure 9, there may be two context trees associated with the bidirectional mixed reference prediction mode, since the computed context does not require a comparison of forward and backward predicted frames of neighboring blocks. Figure 10A shows an example of a context tree 1000 for entropy coding a forward frame identifier of a block encoded using a bidirectional mixed reference prediction mode, and Figure 10B shows an example of a context tree 1002 for entropy coding a backward frame identifier of a block encoded using a bidirectional mixed reference prediction mode.
[0072] The context tree 1000 describes the binary code of past reference frame slots. Each node P, P1, P2 of the context tree is an entropy coded bit with a different probability model (e.g., cumulative distribution function) and context. In other words, each node P, P1, P2 represents a different way of computing the context for a particular binary decision at node P, P1, P2.
[0073] For example, node P may compare the cardinality of forward reference frames LAST and LAST2 used in coding the neighboring block with the cardinality of forward reference frames LAST3 and GOLDEN used in coding the neighboring block. If the cardinality of forward reference frames LAST and LAST2 is smaller than the cardinality of forward reference frames LAST3 and GOLDEN, the context value may be set to 0, if the cardinality of forward reference frames LAST and LAST2 is equal to the cardinality of forward reference frames LAST3 and GOLDEN, the context value may be set to 1, if the cardinality of forward reference frames LAST and LAST2 is larger than the cardinality of forward reference frames LAST3 and GOLDEN, the context value may be set to 2. For example, node P1 may compare the cardinality of forward reference frame LAST used in coding the neighboring block with the cardinality of forward reference frame LAST2 used in coding the neighboring block. If the cardinality of the forward reference frame LAST is less than the cardinality of the forward reference frame LAST2, the context value may be set to 0; if the cardinality of the forward reference frame LAST is equal to the cardinality of the forward reference frame LAST2, the context value may be set to 1; if the cardinality of the forward reference frame LAST is greater than the cardinality of the forward reference frame LAST2, the context value may be set to 2. For example, in node P3, the cardinality of the forward reference frame LAST3 used for coding the adjacent block may be compared with the cardinality of the forward reference frame GOLDEN used for coding the adjacent block. If the cardinality of the forward reference frame LAST3 is less than the cardinality of the forward reference frame GOLDEN, the context value may be set to 0; if the cardinality of the forward reference frame LAST3 is equal to the cardinality of the forward reference frame GOLDEN, the context value may be set to 1; if the cardinality of the forward reference frame LAST3 is greater than the cardinality of the forward reference frame GOLDEN, the context value may be set to 2.
[0074] The context tree 1002 describes the binary code of the future reference frame slots. Each node P, P1, P2 of the context tree is an entropy coded bit with a different probability model (e.g., cumulative distribution function) and context. In other words, each node P, P1, P2 represents a different computation method of the context for a particular binary decision at node P, P1, P2. The context of each node in FIG. 10B can be determined in a similar manner to the nodes in FIG. 10A.
[0075] As can be seen from the above discussion, the use of named references and associated reference mappings makes the determination of the probability model relatively complex. This complex determination is an acceptable trade-off for the increased accuracy of the probability model used to code the reference frame identifiers. However, encoders such as encoder 400 allow for flexible selection of reference frames. Identification of a buffer by a label implies a particular quality or statistics regarding the associated reference frame. Due to the flexibility of the encoder, the actual reference frame associated with the reference frame label may not have the expected quality or statistics. For example, a frame identified as LAST2 may instead correspond to what is described above as LAST3. This may reduce the accuracy of the probability model derived using the context tree. Furthermore, the reference frame management scheme is relatively inflexible and does not allow for additional reference frames to be added without significant changes to the code.
[0076] The reference frame management scheme described herein includes ranking a list of n possible references without using named reference-to-reference mapping. Ranking is described in more detail in FIG. 11 and subsequent figures. FIG. 11 illustrates an example process, technique, or method 1100 for coding one or more reference frames for an inter-predicted block. More specifically, the method 1100 describes coding a syntax element representing one or more reference frames.
[0077] First, the method of FIG. 11 assumes that an inter-predicted block is the current block in the coding sequence. For example, when the method 1100 is performed as part of an encoding process, such as with the encoder 400, the current block, the block (if any) preceding the current block in the block coding order of the current frame, and any previous frame(s) in the frame coding order (e.g., blocks of that previous frame) have already undergone inter prediction, transformation, quantization, and reconstruction, such as the reconstruction pass of FIG. 4, if applicable. Thus, the method 1100 can be implemented in whole or in part during entropy coding, such as with the entropy encoding stage 408 of the encoder 400 of FIG. 4. In another example, when the method 1100 is performed as part of a decoding process, such as with the decoder 500, the current block, the block (if any) preceding the current block in the block coding order of the current frame, and any previous frame(s) in the frame coding order (e.g., blocks of that previous frame) have been received as part of an encoded bitstream, such as the compressed bitstream 420, the previously coded blocks have been decoded, and the process to decode the current block has begun. In this latter example, the method 1100 may be implemented in whole or in part during entropy coding, such as entropy decoding stage 502 of decoder 500 of FIG.
[0078] The method 1100 can be implemented as a software program that can be executed by a computing device, such as the transmitting station 102 or the receiving station 106 of FIG. 1. For example, the software program includes machine-readable instructions that can be stored in a memory, such as the memory 204 or the secondary storage 214 of FIG. 2, and that, when executed by a processor, such as the CPU 202 of FIG. 2, cause the computing device to perform the method 1100. The method 1100 can be implemented using dedicated hardware or firmware. Some computing devices may include multiple memories or processors, and the operations described in the method 1100 can be distributed using multiple processors, memories, or both. The method 1100 can be implemented using dedicated hardware or firmware. Multiple processors, memories, or both can be used.
[0079] At 1102, the method 1100 determines whether the coding mode used to code the current inter-predicted block is a single reference coding mode or a mixed reference coding mode. A syntax element to be coded is then identified at 1104. In the example described herein, the syntax element is an identifier of a reference frame or an identifier of a reference frame used to code the current block. In an encoder, this information may be passed to the entropy encoding stage and embedded in the header of the current block. In a decoder, this information may be read from the header of the current block at the entropy decoding stage.
[0080] At 1106, context information for coding the syntax elements may be identified. The context information may be syntax elements associated with previously coded blocks. For example, the previously coded blocks may include one or more blocks that are temporally adjacent to the current block in a previously coded frame, one or more blocks that are spatially adjacent to the current block in the current frame, or some combination thereof. The context information may include a value that identifies a respective inter-coding mode (e.g., single reference or mixed reference) and a reference frame used to code the previously coded block. The context information may be stored in and retrieved from a cache or other memory.
[0081] At 1108, the node of the context tree containing the syntax element is determined. In the above example, a selection must be made from multiple context trees to use the named reference assignment / slot. A frame-level reference syntax for such a scheme includes, for example, an 8-bit flag indicating which buffer(s) of the reference frame buffers, such as reference frame buffer 600, to update with the current frame. The frame-level reference syntax also includes a list of values indicating the mapping of buffers to the named reference assignments. For example, 3 bits each may be used to map a buffer to each of, for example, 7 named reference slots. These index values may be coded as literals.
[0082] The ranked reference framework described herein allows the refresh bitmap and reference mapping to be derived not only on the encoder side but also on the decoder side. This removes the need to directly signal these values. Furthermore, the ranked reference framework significantly simplifies the context tree. It is worth noting that the ranked reference framework described herein does not need to completely replace the named reference buffer system in all implementations. For example, different frame groups of the same video stream being encoded or decoded can use one or the other system. This can be achieved, for example, by using a flag encoded in the header of the key frame indicating whether the ranking of reference frames is enabled.
[0083] As mentioned at the beginning, the ranked reference framework uses heuristics designed to determine the usefulness of each possible reference frame, its likelihood of being a good choice for predicting a block in the current frame (e.g., its likelihood of minimizing the residual of the block), or some combination thereof. For example, studies on encoded blocks have shown that temporal distance from the current frame is generally a strong indicator of the usefulness of a reference frame for prediction. Thus, in some implementations, the ranking may consider temporal distance (i.e., display order distance between the reference and the current frame) as one of several frame statistics used to score the reference frames for the ranking. Other statistics may include, but are not limited to, the quality or quantization level of the frame, the number of times the frame has been selected for reference, gradient information of the frame, or some combination thereof. For example, studies on encoded blocks have shown that quality or quantization level of the frame is generally a strong indicator of the usefulness of a reference frame for prediction. In some implementations, the quality level may be useful when multiple references have the same or similar temporal distance. Additionally, the quality may be useful when multiple references are in the past (e.g., more than three frames apart).
[0084] Each of the possible reference frames may be ranked using, for example, a combination of differences between the respective statistics of the reference frame and the current frame, a combination of differences between the respective statistics of the reference frames, or some combination thereof. For example, if the first reference frame is closer to the current frame in display order, the first reference frame may be ranked higher than the second reference frame. For example, if the first reference frame is of higher quality than the second reference frame, the first reference frame may be ranked higher than the second reference frame. For example, if the gradient information of the first reference frame is closer to the gradient information of the current frame than the gradient information of the second reference frame, the first reference frame may be ranked higher than the second reference frame. For example, if the first reference frame is used more than the second reference frame, the first reference frame may be ranked higher than the second reference frame.
[0085] In one implementation, the distance between the current frame and the reference frame may be used first for ranking. As long as the reference frame is the same distance from the current frame, the next metric can be used for ranking. Here, a higher value of the variable score indicates that the reference frame is more likely to be useful for inter prediction than a lower value of the variable score.
[0086]
number
[0087] In the metrics that determine the score above, D C is the display order number of the current frame, D R is the display sequence number of the reference frame, L R is the quality level of the reference frame, L C is the quality level of the current frame. Since the absolute value of the difference in display order numbers is determined, it is irrelevant whether the reference frame is before or after the current frame in display order.
[0088] With regard to the quality level, several values associated with a frame may be used to indicate the quality level of the frame. In some implementations, the quality level may be determined by the quantization value or quantization level of the frame. As described with respect to FIG. 4, a quantization stage, such as quantization stage 406, divides the transform coefficients of a frame block by the quantization value and truncates the resulting value before entropy encoding. The higher the quantization value or quantization level, the lower the quality level of the frame. Conversely, the lower the quantization value or quantization level, the higher the quality level of the frame. Thus, in some implementations, the quality level may be the inverse of the quantization value or quantization level, as in the above metric for determining the score. The quality level of the reference frame L R is the quality level of the current frame L C If higher than, the score value is the quality level of the reference frame L R is the quality level of the current frame L C If it is lower, it will be higher.
[0089] In another implementation, the distance between the current frame and the reference frame may be used first for ranking, and if the reference frame is the same distance from the current frame, the next metric may be used for ranking. Again, a higher value of the variable score indicates that the reference frame is more likely to be useful for inter prediction than a lower value of the variable score.
[0090]
number
[0091] In an example of applying Equation (1) and Equation (2), if three reference frames are available, one of which is located next to the current frame and two of which are one frame away from the current frame, the adjacent reference frame is ranked highest, and the remaining two reference frames are given scores calculated according to Equation (1) or Equation (2). The one with the higher score is ranked second, and the other is ranked third.
[0092] In yet another embodiment, the following metrics may be used to rank the reference frames:
[0093]
number
[0094] In this metric, Q R is the quality level of the reference frame. The score may be determined by taking a function of the absolute value of the difference in the display order numbers. In some implementations, the function may be a concave function. For example, the function f(x) may be a monotonically increasing concave function as shown in the following table:
[0095] [Table 1]
[0096] In this metric, a lower value of the variable score indicates that the reference frame is more likely to be useful for inter prediction than a higher value of the variable score. R Regarding L R Similarly, some value associated with the frame may be used to indicate the quality level of the reference frame. In some implementations, the quality level may be determined by the quantization value or quantization level of the reference frame, as described above. However, L in Equations (1) and (2) may be used to indicate the quality level of the reference frame. R In contrast, in equation (3), the quality level Q Ris high for low quality frames and low for high quality frames. Therefore, Q R The value of may be the quantization value or quantization level itself, or a representative value thereof, such as the base frame value used to code other quantizers (e.g., as opposed to the inverse of this value or level). The ranking may be completed using, for example, a simple bubble sort.
[0097] In yet another embodiment, the following metrics may be used to rank the reference frames:
[0098]
number
[0099] Similar to equation (3), a lower value of the variable score indicates that the reference frame is more likely to be useful for inter prediction than a higher value of the variable score. The ranking can be completed using, for example, a simple bubble sort.
[0100] According to the above, temporal distance is a strong indicator of the usefulness of a reference frame. Apart from the temporal distance, the quality of the reference frame may be considered, especially when multiple references have the same or similar temporal distance from the current frame. In some implementations, the number of past reference frames compared to the total number of frames may be relevant. For example, in low latency configurations or when encoding forward key frames with random access, multiple reference frames may be far in the past (e.g., more than 3 frames away). This makes quality more important for such frames, and it may be desirable to calculate the score differently depending on the cardinality of the past reference frames and / or the cardinality of the future reference frames. In one implementation, both equation (3) and equation (4) may be used. For example, equation (3) may be used to calculate the score when all reference frames are in the past, and equation (4) may be used to calculate the score otherwise.
[0101] These same principles may be useful when the number of available reference frames is greater than the number of reference slots, e.g., 8 reference frames are available for 7 active reference slots. Deciding which reference frame to use may involve calculating the average quality level of all available reference frames (e.g., Q R If there are more past reference frames than future reference frames, the quality level indicates a lower quality frame than the average (e.g., quality level Q R The lowest-scoring (e.g., highest-scoring) past reference frame is omitted. Otherwise, the quality level indicates a lower-than-average quality frame (e.g., quality level Q R The future reference frame with the lowest score (where the average quality level is exceeded) is omitted.
[0102] Once a frame block of a frame group is reconstructed, the reconstructed frame becomes available as a reference frame for coding further frames. Each reference frame available for inter prediction of a block of the current frame may be considered using a ranking up to the maximum number of reference frames available per frame. When encoding or decoding a group of frames, upon receipt of the first (e.g., key) frame, all reference frame buffers may be updated. At the next frame of the coding sequence, the subsequent frame becomes available for inter prediction of that block, and so on. As the reconstructed frame becomes available for reference, it is added to a buffer such as reference frame buffer 600, and scores are recalculated to map the reference frames to ranks.
[0103] In some implementations, ranked reference frames can be used to speed up encoder decisions by excluding reference frames from the search for optimal prediction modes using rate-distortion techniques. For example, in the case of mixed prediction, if two reference frames have ranks 4 or higher (i.e., ranks 4, 5, 6, 7), combinations using both reference frames may not be considered by the encoder. As mentioned above, the direction of the reference frames may be relevant for coding decisions. Thus, the reference frames are ranked separately in directional lists, one for past reference frames and one for future reference frames, and each frame in each list is given a directional rank based on the value of the score. If the prediction of the mixed prediction mode of the current block is unidirectional, combinations with both directional ranks greater than 2 may be omitted. Otherwise (if the prediction is bi-predictive), combinations with both reference frames with directional ranks greater than 3 may be omitted. A special case may be when all references are past. In such a situation, combinations with both directional ranks greater than 4 may be omitted, and more searches are performed than in the general case.
[0104] As mentioned above, the ranked reference framework simplifies the determination of the context. One reason for this simplified determination is that the ranked reference framework does not distinguish between past and future reference frames. Therefore, the same context tree may be used for each reference frame used to code a block. In some implementations, the same syntax structure may be used to code reference frame identifiers for both single and mixed reference prediction modes. As shown in FIG. 12, an example of a syntax structure of a context tree 1200 that can be used to code identifiers in which the ranked reference framework is implemented.
[0105] The context tree 1200 of FIG. 12 can be used to explain how the nodes of the context tree containing syntax elements are determined in 1108. The context tree 1200 is a binary tree that determines the Rice code for entropy coding. In the context tree 1200, the highest ranked reference frame is labeled REF RANK1, the next highest ranked reference frame is labeled REF RANK2, the next highest ranked reference frame is labeled REF RANK3, and so on until the lowest ranked reference frame, i.e., REF RANK7, is labeled. For example, because the context tree 1200 is a binary tree, the value of the context information can be resolved as true (1) or false (0) for the separation criteria at each tree level. This example assumes that there are seven reference frames available, although fewer or additional reference frames may be used.
[0106] First, the single reference prediction mode is described. At node P0, it is determined whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK1. If so, code 1 is used for entropy coding. If not, at node P1, it is considered whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK2. If so, code 01 is used for entropy coding. If not, at node P2, it is considered whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK3. If so, code 001 is used for entropy coding. If not, at node P3, it is considered whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK4. If so, code 0001 is used for entropy coding. If not, at node P4, it is considered whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK5. If so, the code 00001 is used for entropy coding. If not, then in node P5 it is considered whether the reference frame used for coding the current block corresponds to the one identified by the label REF RANK6. If so, the code 000001 is used for entropy coding. If not, then the reference frame used for coding the current block corresponds to the one identified by the label REF RANK7 and the code 000000 is used for entropy coding.
[0107] At 1110, syntax elements are coded according to a probability model using the context information associated with the nodes determined at 1108. A cumulative distribution function (e.g., obtained from the probability model for each node) can be determined using a context value that includes the code of the node. A context value is determined for each node, and the context value may be determined using the context information identified at 1106. For example, for node P0, if the cardinality of the reference frame used by the neighboring block identified as REF RANK1 is less than the sum of each respective cardinality of each reference frame identified as REF RANK2 through REF RANK7, the context value is 0, if the cardinality of the reference frame used by the neighboring block identified as REF RANK1 is equal to said sum, the context value is 1, and if the cardinality of the reference frame used by the neighboring block identified as REF RANK1 is greater than said sum, the context value is 2. For node P1, if the cardinality of the reference frames used by the neighboring blocks identified as REF RANK2 is less than the sum of the respective cardinality of each of the reference frames identified as REF RANK3 to REF RANK7, the context value is 0, if the cardinality of the reference frames used by the neighboring blocks identified as REF RANK2 is equal to said sum, the context value is 1, and if the cardinality of the reference frames used by the neighboring blocks identified as REF RANK2 is greater than said sum, the context value is 2. More generally, for each node, the context value of each node is determined according to the following table, where N is the total number of ranked references and n is the ranking of the reference frames associated with the current node, n=1 to N-1.
[0108] [Table 2]
[0109] In some implementations, the reference frame context information identified in 1106 and used to determine the context value may be limited to context information from neighboring blocks predicted using a single reference prediction mode.
[0110] Each node of the context tree 1200 is associated with a probability model. The probability model associated with a node can reflect the probability of the syntax element of that node. The probability model can indicate the probability that the syntax element associated with a current block of a video frame will have a particular value (e.g., the probability that the reference frame used to predict the current block is a particular reference frame). In one example, the probability model can include integer values reflecting different probabilities that may be associated with each bit of the code representing the syntax element. A maximum value can be defined for the probability model, and a given probability can be expressed as a percentage obtained by dividing the integer value by the maximum value.
[0111] During an encoding operation, entropy coding (e.g., Golomb-Rice coding) is performed on the syntax element according to the identified probability model, and the resulting code is then included in the current block header in the encoded bitstream. Alternatively, during a decoding operation, the coded syntax element is obtained from the current block header in the encoded bitstream, arithmetic coding is performed on the syntax element according to the identified probability model, reference frame(s) are identified, and the encoded block is decoded.
[0112] Returning to 1108, the determination of the nodes of each syntax element (e.g., respective identifiers of reference frames) for which a mixed reference prediction mode is used can be performed at 1108 using the same syntax structure as the context tree 1200, but a different cumulative distribution function is used for the nodes for coding at 1110. The lowest ranked reference frame can be coded first in the same manner as described above for the single reference frame case. The second reference frame can be coded by continuing down the syntax tree 1200 from the node of the first reference frame. For example, if the mixed reference prediction mode of the current block uses {REF RANK2, REF RANK5}, the code 01001 is used for entropy coding. The reference frame context information used to determine the context value may be limited to context information from neighboring blocks predicted using the mixed reference prediction mode.
[0113] The method according to FIG. 11 can be performed for each block of the current frame being coded using inter prediction.
[0114] Although each of these examples describes using a neighboring block (or neighboring blocks) to identify context information at 1106, the teachings herein are not limited to this approach. For example, when blocks of a frame are coded in coding order (e.g., raster scan order), inter-predicted blocks can be used to accumulate context information.
[0115] In addition to simplifying entropy coding, the use of a ranked reference framework offers the added benefit of eliminating the frame headers (i.e., refresh bitmaps and reference mappings) required for coding reference frames. For example, the use of ranked reference lists can speed up encoding algorithms by allowing the encoder to skip mode searches for reference frames that are lower ranked in the list.
[0116] In the above example ranked reference framework, seven reference frames are used. However, since the determination of the nodes in the context tree and the determination of the context values are substantially similar, it is relatively easy to handle fewer or additional reference frames than the seven described. The ranked reference framework allows flexibility in the number of references available per frame and per block. This can be achieved by signaling (e.g., in the frame header) the maximum number of reference frames to include in the ranking for each frame. Alternatively, or additionally, block level syntax may be improved in cases where not all available references of a particular frame are used. In this case, for example, the frame header may include syntax such as a mask indicating which reference frames a block may use. For example, a 7-bit mask may indicate the use of reference frame (1) and the use of reference frame (0). In some implementations, the segment or block header may include syntax such as a mask indicating which reference frames a segment or block may use, similar to that described for the frame header. Reference frames may be removed if they are not useful, for example, one block of a frame may be limited to four reference frames and another block to two reference frames.
[0117] The new reference framework described here ranks reference frames based on a normative procedure (e.g., a calculated score) and signals the reference frames based on their rank. The bitstream syntax is simplified by using a context tree that depends on the ranking. Furthermore, mapping reference frames to buffers can be determined at the decoder without the need for signaling.
[0118] For ease of explanation, methods described herein, such as method 1100 of FIG. 11 and variations thereof, are illustrated and described as a series of steps or operations. However, steps or operations in accordance with the present disclosure may be performed in various orders and / or simultaneously. Moreover, other steps or operations not shown and described herein may be used. Moreover, not all illustrated steps or operations are required to implement a methodology in accordance with the disclosed subject matter.
[0119] The encoding and decoding aspects described above provide some examples of encoding and decoding techniques, although it should be understood that encoding and decoding, as the terms are used in the claims, may mean compressing, decompressing, converting, or other data processing or modification.
[0120] The word "example" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as an "example" should not necessarily be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any of the natural inclusive permutations. That is, if X includes A, if X includes B, or if X includes both A and B, then "X includes A or B" is satisfied in any of the foregoing cases. Furthermore, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless otherwise specified or clear from the context to refer to the singular form. Additionally, use of the term "embodiment" or "one embodiment" throughout is not intended to refer to the same embodiment or implementation, unless specifically stated to that effect.
[0121] The implementation of the sending station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc. stored therein and / or executed by, also executed by the encoder 400 and the decoder 500) can be realized by hardware, software, or any combination thereof. Hardware includes, for example, computers, intellectual property (IP) cores, application specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuitry. In the claims, the term "processor" should be understood to include any of the above hardware, alone or in combination. The terms "signal" and "data" are used interchangeably. Furthermore, each part of the sending station 102 and the receiving station 106 does not necessarily have to be implemented in the same way.
[0122] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 may be implemented using a general purpose computer or processor with a computer program that, when executed, performs any of the respective methods, algorithms, and / or instructions described herein. Additionally or alternatively, a special purpose computer / processor may be utilized that may include, for example, other hardware for performing any of the methods, algorithms, or instructions described herein.
[0123] The sending station 102 and the receiving station 106 can be implemented on computers, for example, in a videoconferencing system. Alternatively, the sending station 102 can be implemented on a server and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communication device. In this case, the sending station 102 can encode content into an encoded video signal using the encoder 400 and transmit the encoded video signal to the communication device. The communication device can then decode the encoded video signal using the decoder 500. Alternatively, the communication device can decode content stored locally on the communication device, for example, content that was not transmitted by the sending station 102. Other suitable sending and receiving implementation schemes are also available. For example, the receiving station 106 can be a generally fixed personal computer rather than a portable communication device, and / or the device including the encoder 400 can include the decoder 500.
[0124] Furthermore, all or part of the embodiments of the present disclosure may take the form of a computer program product accessible, for example, from a computer usable or computer readable medium. A computer usable or computer readable medium may be, for example, any device that can tangibly store, preserve, communicate, or transfer a program for use by or in connection with any processor. The medium may include, for example, electronic, magnetic, optical, electromagnetic, or semiconductor devices. Other suitable media may also be utilized.
[0125] The above-mentioned embodiments, aspects and aspects are described to facilitate understanding of the present invention, and are not intended to limit the present invention. Rather, the present invention is intended to cover various modifications and equivalent structures included within the scope of the appended claims, and the scope should be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as permitted by law.
Claims
1. 1. A method for coding an identifier of at least one reference frame used for coding a current block of a video frame, comprising the steps of: identifying a syntax element to be coded, the syntax element corresponding to the identifier; determining context information for the syntax element, the context information including a reference frame used to code a block preceding the current block in coding order among available reference frames; determining a node of a context tree that includes the syntax element, the context tree being a binary tree that includes as nodes the available reference frames arranged in a ranking based on a score calculated for each reference frame using at least one characteristic of the reference frame that indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames; and coding the syntax element according to a probabilistic model using the context information associated with the node.
2. The method of claim 1 , wherein the at least one characteristic comprises at least two different characteristics.
3. 3. The method of claim 1, further comprising: determining whether a coding mode of the current block is a single reference coding mode or a mixed reference coding mode, wherein the context tree has the same syntactic structure regardless of whether the coding mode of the current block is the single reference coding mode or the mixed reference coding mode.
4. The method of claim 1 , wherein the available reference frames included in the context tree include forward and backward reference frames.
5. the context tree is a single context tree including a forward reference frame and a backward reference frame; the coding mode of the current block is a mixed reference coding mode; the identifiers include a first identifier for a first reference frame and a second identifier for a second reference frame; identifying the syntax element to be coded includes identifying a first syntax element corresponding to the first identifier and identifying a second syntax element corresponding to the second identifier; determining the nodes includes determining a first node of the single context tree that includes the first syntax element and determining a second node of the single context tree that includes the second syntax element; determining the context information for the syntax elements includes determining first context information for the first syntax element and determining second context information for the second syntax element; 3. The method of claim 1, wherein coding the syntax element comprises coding the first syntax element according to a first probability model using the first context information, and coding the second syntax element according to a second probability model using the first context information.
6. 6. The method of claim 1, wherein the at least one characteristic comprises a quality level of the reference frame of the available reference frames being ranked and a temporal distance of the reference frame from the video frame containing the current block.
7. 2. The method of claim 1, wherein a frame header of the video frame including the current block includes a mask having a bit length equal to a cardinality of the available reference frames, the mask indicating which of the available reference frames the current block may use for inter prediction and which of the available reference frames the current block may not use for inter prediction.
8. 1. A method for decoding a current block of a video frame, comprising the steps of: ranking the list of available reference frames according to at least one characteristic of each reference frame; decoding, from a block header of the current block, an identifier of at least one reference frame used to encode the current block using a context tree, the context tree being a single binary tree having nodes arranged from a highest-ranked reference frame of the available reference frames to a lowest-ranked reference frame of the available reference frames; and decoding the current block using the at least one reference frame.
9. The method of claim 8 , wherein the at least one characteristic of each reference frame indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames.
10. 10. The method of claim 8 or 9, wherein the at least one characteristic comprises at least two of: a temporal distance from the video frame containing the current block, a quality or quantization level of the reference frame, a number of times the reference frame has been selected for inter prediction, or gradient information of the reference frame.
11. 1. A method for encoding a current block of a video frame, comprising the steps of: ranking the list of available reference frames based on a score calculated for each reference frame using at least one characteristic of the reference frame that indicates whether the reference frame is likely to encode the current block more efficiently than other reference frames; encoding the current block using at least one of the available reference frames; and and encoding in a block header of the current block an identifier of at least one reference frame used to encode the current block using a context tree, the context tree being a single binary tree having nodes arranged from a highest-ranked reference frame of the available reference frames to a lowest-ranked reference frame of the available reference frames.
12. encoding the current block, coding the current block multiple times using a single reference coding mode and each of the available reference frames except for at least the lowest ranked reference frame; and selecting the at least one reference frame as the single reference frame having the highest coding efficiency of the available reference frames.
13. 12. The method of claim 11, wherein encoding the current block using at least one reference frame includes encoding the current block using two reference frames, and encoding the identifier includes encoding a first identifier of a first reference frame of the two reference frames using the single binary tree and encoding a second identifier of a second reference frame of the two reference frames using the single binary tree.
14. The method according to any of claims 11 to 13, wherein the available reference frames of the single binary tree include a forward reference frame and a backward reference frame.
15. Apparatus comprising a processor configured to carry out the method of any of claims 1 to 14.
Citation Information
Patent Citations
Coding video syntax elements using context trees
JP2020523862A
