Apparatus, method and computer program for video encoding and decoding

Abstract

A method comprising: determining a list of context classes; determining a first superblock of a picture; determining a first context class from the list of context classes; determining a first bin value belonging to the first context class in the first superblock; using the first bin value to calculate a value for a first probability estimate; encoding a syntax element using the first probability estimate; determining a second superblock of the picture; determining a second context class from the list of context classes; determining a second bin value belonging to the second context class in the second superblock; using the second bin value to calculate a value for a second probability estimate; and encoding the syntax element using the second probability estimate.

Description

Technical Field

[0001] This invention relates to apparatus, methods, and computer programs for video encoding and decoding. Background Technology

[0002] Arithmetic encoders, such as Context Adaptive Binary Arithmetic Encoders (CABAC), are used in video coding to efficiently compress and decompress different syntax elements. In the case of Context Adaptive Arithmetic Encoding, each syntax element has a probability estimate for its possible values. These probability estimates are called contexts because they provide a context or model describing the behavior of one or more syntax elements associated with that context. A list of context categories identifies the context categories that have been selected for spatial tuning of the state of that arithmetic encoder. In CABAC, a separate list of coding context categories is used for each superblock, such as a Code Tree Unit (CTU).

[0003] The encoding and decoding of image and video content are typically performed within a coding unit (CTU) in a nested tree order, starting from the top-left corner of the CTU and ending at the bottom-right corner. As a new CTU is reached, some memory is needed to store the previously computed syntax element values ​​in different neighborhoods of the image, along with a list of the coding context categories of the corresponding superblocks, in order to adjust the probabilities of the arithmetic coding engine towards their optimal values ​​in the local neighborhood of the syntax element to be encoded or decoded.

[0004] It is easy to conclude that the amount of such data can be relatively large, and it needs to be buffered over the time required for encoding / decoding across multiple CTUs. Efficiently storing such a large amount of data in a way that minimizes the amount of memory required for storage devices presents various challenges. Summary of the Invention

[0005] Now, in order to at least alleviate the above problems, this paper introduces an enhancement method.

[0006] The scope of protection sought by the various embodiments of this invention is defined by the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims (if any) should be interpreted as examples useful for understanding the various embodiments of the invention.

[0007] The method according to the first aspect includes: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; using the first bin value to calculate a value for a first probability estimate; encoding syntax elements using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; using the second bin value to calculate a value for a second probability estimate; and encoding syntax elements using the second probability estimate.

[0008] The method according to the second aspect includes: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; using the first bin value to calculate a value for a first probability estimate; decoding the syntax element using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; using the second bin value to calculate a value for a second probability estimate; and decoding the syntax element using the second probability estimate.

[0009] The apparatus according to a third aspect includes: components for determining a list of context categories; components for determining a first superblock of an image; components for determining a first context category from the list of context categories; components for determining a first bin value belonging to the first context category in the first superblock; components for calculating a value for a first probability estimate using the first bin value; components for encoding syntax elements using the first probability estimate; components for determining a second superblock of an image; components for determining a second context category from the list of context categories; components for determining a second bin value belonging to the second context category in the second superblock; components for calculating a value for a second probability estimate using the second bin value; and components for encoding syntax elements using the second probability estimate.

[0010] The apparatus according to the fourth aspect includes: components for determining a list of context categories; components for determining a first superblock of an image; components for determining a first context category from the list of context categories; components for determining a first bin value belonging to the first context category in the first superblock; components for calculating a value for a first probability estimate using the first bin value; components for decoding a syntax element using the first probability estimate; components for determining a second superblock of an image; components for determining a second context category from the list of context categories; components for determining a second bin value belonging to the second context category in the second superblock; components for calculating a value for a second probability estimate using the second bin value; and components for decoding a syntax element using the second probability estimate.

[0011] According to one embodiment, the apparatus includes components for updating a list of context categories at the beginning of an image, at the beginning of a slice, or at the beginning of a superblock line.

[0012] According to one embodiment, the list of context categories has a defined maximum size.

[0013] According to one embodiment, the apparatus includes a component for constructing a list of context categories by including context categories from at least two superblocks.

[0014] According to one embodiment, the apparatus includes a component for constructing a list of context categories by including at most a determined maximum number of context categories from a single superblock.

[0015] According to one embodiment, the apparatus includes components for constructing a context category list having a determined number of context categories calculated using an index of an encoding unit.

[0016] According to one embodiment, the apparatus includes a component for storing bin values ​​associated with context categories included in a list of context categories in a superblock-specific buffer.

[0017] According to one embodiment, the apparatus includes: components for updating at least a first probability estimate by determining a first context category using a list of context categories and stored bin values ​​for a first superblock; and components for updating at least a second probability estimate by determining a second context category using a list of context categories and stored bin values ​​for a second superblock.

[0018] According to one embodiment, the apparatus includes: a component for updating at least one probability estimate associated with a first context category in a first superblock based on stored bin values ​​of a second superblock; and a component for updating at least one probability estimate associated with a second context category in a third superblock based on stored bin values ​​of a fourth superblock, wherein the first context category and the second context category are determined using the same list of context categories.

[0019] According to one embodiment, the apparatus includes components for sharing two lists of context categories among a plurality of superblocks, wherein a first list of context categories is used to store context categories, bin values ​​for context categories are stored for updating probability estimates for the next superblock row, and a second list of context categories is used to identify which context categories are stored on the previous superblock row and is used for updating probability estimates for the current superblock row.

[0020] According to one embodiment, the apparatus includes: a component for resetting a first context category list before processing a first superblock in a superblock row; and a component for skipping the reset of a second context category list.

[0021] According to one embodiment, a superblock is a coding tree unit (CTU), a maximum coding unit (LCU), or a macroblock.

[0022] As a fifth aspect, an apparatus is provided, comprising: at least one processor and at least one memory, wherein code is stored on the at least one memory, the code, when executed by the at least one processor, causes the apparatus to perform at least: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; calculating a value for a first probability estimate using the first bin value; encoding a syntax element using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; calculating a value for a second probability estimate using the second bin value; and encoding a syntax element using the second probability estimate.

[0023] As a sixth aspect, an apparatus is provided, comprising: at least one processor and at least one memory, wherein code is stored on the at least one memory, the code, when executed by the at least one processor, causes the apparatus to perform at least: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; calculating a value for a first probability estimate using the first bin value; decoding a syntax element using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; calculating a value for a second probability estimate using the second bin value; and decoding a syntax element using the second probability estimate.

[0024] Therefore, the apparatus described above and the computer-readable storage medium thereon storing code are arranged to perform the methods described above and one or more embodiments thereto. Attached Figure Description

[0025] To better understand the invention, reference will now be made to the accompanying drawings by way of example, in which:

[0026] Figure 1a and Figure 1b Accordingly, an encoder and decoder suitable for implementing embodiments of the present invention are schematically shown;

[0027] Figure 2 This illustrates a typical segmentation of an image into a coding tree unit (CTU) or superblock;

[0028] Figure 3 An example of nested tree encoding / decoding order of encoding units (CUs) within an encoding tree unit (CTU) is shown;

[0029] Figure 4 A flowchart of an encoding method according to one embodiment is shown;

[0030] Figure 5 A flowchart of a decoding method according to one embodiment is shown;

[0031] Figure 6 An example memory array layout according to one embodiment is shown;

[0032] Figure 7 An electronic device suitable for employing embodiments of the present invention is illustrated schematically;

[0033] Figure 8 A user equipment suitable for employing embodiments of the present invention is illustrated schematically;

[0034] Figure 9 The electronic device, which is suitable for employing embodiments of the present invention, is further illustrated schematically, and is connected using wireless and wired network connections;

[0035] Figure 10 A schematic diagram of an example multimedia communication system in which various embodiments can be implemented is shown. Detailed Implementation

[0036] A video codec consists of an encoder and a decoder. The encoder transforms the input video into a compressed representation suitable for storage / transmission, and the decoder decompresses the compressed video representation back into a viewable form. Video encoders and / or video decoders can also be separate entities, meaning they don't necessarily form a codec. Typically, the encoder discards some information from the original video sequence to represent the video in a more compact form (i.e., at a lower bit rate).

[0037] Typical hybrid video encoders (such as many encoder implementations of ITU-T H.263 and H.264) encode video information in two stages. First, pixel values ​​in a picture region (or “block”) are predicted, for example, by motion compensation (finding and indicating the region closely corresponding to the block being encoded in one of the previously encoded video frames) or by spatial methods (using the pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically done by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0038] In temporal prediction, the source of the prediction is a previously decoded image (also called a reference image). In intra-block copy (IBC; also known as intra-block copy prediction), prediction is applied in a similar manner to temporal prediction, but the reference image is the current image, and only previously decoded samples can be referenced during the prediction process. Inter-layer prediction or inter-view prediction can be applied in a similar manner to temporal prediction, but the reference image is, respectively, a decoded image from another scalable layer or from another view. In some cases, inter-frame prediction may refer only to temporal prediction, while in others, inter-frame prediction may be collectively referred to as temporal prediction as well as any of the following: intra-block copy, inter-layer prediction, and inter-view prediction, provided they are performed using the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensation prediction.

[0039] Motion compensation can be performed with full-sample precision or subsample precision. In the case of full-sample precision motion compensation, motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses these displacements to effectively replicate samples from a reference image. In the case of subsample precision motion compensation, the motion vector is represented by fractional or decimal values ​​for the horizontal and vertical components of the motion vector. When the motion vector points to a non-integer position in the reference image, a subsample interpolation process is typically invoked to compute predicted sample values ​​based on the reference sample and the selected subsample position. The subsample interpolation process typically consists of horizontal filtering to compensate for horizontal offsets relative to the full-sample position, followed by vertical filtering to compensate for vertical offsets relative to the full-sample position. However, in some environments, vertical processing may also be performed before horizontal processing.

[0040] Inter-frame prediction (also known as temporal prediction, motion compensation, or motion-compensated prediction) reduces temporal redundancy. In inter-frame prediction, the prediction source is the previously decoded image. Intra-frame prediction leverages the fact that adjacent pixels within the same image may be related. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, sample values ​​or transform coefficients can be predicted. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0041] One outcome of the encoding process is a set of encoded parameters, such as motion vectors and quantization transform coefficients. If many parameters are first predicted from spatially or temporally adjacent parameters, they can be entropy-coded more efficiently. For example, motion vectors can be predicted from spatially adjacent motion vectors, and only the difference relative to that motion vector predictor can be encoded. The prediction of encoded parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0042] Figure 1a and Figure 1b An encoder and decoder suitable for employing embodiments of the present invention are illustrated. The video codec consists of an encoder that transforms the input video into a compressed representation suitable for storage / transmission, and a decoder that decompresses the compressed video representation back to a viewable form. Typically, the encoder discards and / or loses some information from the original video sequence to represent the video in a more compact form (i.e., at a lower bit rate). An example of the encoding process is shown in... Figure 1a It is shown in the middle. Figure 1a The image to be encoded (I) is shown. n ); Predictive representation of image patches (P' n ); Prediction error signal (D) n ); reconstructed prediction error signal (D' n ); Preliminary image reconstruction (I' n); final reconstructed image (R' n ); Transform (T) and Inverse Transform (T) -1 ); Quantization (Q) and dequantization (Q) -1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P) inter Intra-frame prediction (P) intra ); Mode selection (MS) and filtering (F).

[0043] An example of the decoding process is in Figure 1b It is shown in the middle. Figure 1b The predicted representation of the image patch (P') is shown. n ); reconstructed prediction error signal (D' n ); Preliminary image reconstruction (I' n ); final reconstructed image (R' n ); Inverse Transform (T) -1 Inverse quantization (Q) -1 ); Entropy decoding (E -1 ); Reference Frame Memory (RFM); Prediction (inter-frame or intra-frame) (P); and Filtering (F).

[0044] Many hybrid video encoders encode video information in two stages. First, pixel values ​​in a picture region (or “block”) are predicted, for example, by motion compensation (finding and indicating the region closely corresponding to the block being encoded in one of the previously encoded video frames) or by spatial methods (using pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically done by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate). Video codecs can also provide a transform skip mode, which the encoder can optionally use. In transform skip mode, the prediction error is encoded in the sample domain, for example, by deriving a sample-by-sample difference relative to some neighboring samples and encoding that sample-by-sample difference using an entropy encoder.

[0045] Entropy coding / decoding can be performed in various ways. For example, context-based coding / decoding can be applied, where the context state of the coding parameters is modified based on the previously encoded / decoded coding parameters in both the encoder and decoder. Context-based coding can be, for example, Context Adaptive Binary Arithmetic Coding (CABAC), Context-Based Variable-Length Coding (CAVLC), or any similar entropy coding. Entropy coding / decoding can be performed alternatively or additionally using variable-length coding schemes, such as Huffman coding / decoding or Exp-Golomb coding / decoding. Decoding the coding parameters from the entropy-coded bitstream or codewords can be called parsing.

[0046] The term "along the bitstream" (e.g., indicating along the bitstream) can be defined as referring to out-of-band transmission, signaling, or storage in a way that associates out-of-band data with the bitstream. The term "decoding along the bitstream" or similar expressions can refer to decoding the indicated out-of-band data (which can be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream. For example, an indication along the bitstream can refer to metadata in a container file encapsulating the bitstream.

[0047] The H.264 / AVC standard was developed by the Video Coding Experts Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by both parent standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also referred to as MPEG-4 Part 10 Advanced Video Coding (AVC). Several versions of the H.264 / AVC standard have been released, integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0048] Version 1 of the High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaboration Group on Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by two parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also referred to as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multiple View, Fidelity Range, 3D, and Screen Content Coding Extensions, which can be abbreviated accordingly as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC.

[0049] Multifunctional Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Group (JVET) of the Moving Picture Experts Group (MPEG) (formally ISO / IEC JTC1 SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU), as a successor to HEVC / H.265.

[0050] This section describes some key definitions, bitstreams, and encoding structures and concepts of H.264 / AVC and HEVC to provide background for video encoders, decoders, encoding methods, decoding methods, and bitstream structures in which embodiments can be implemented. Some key definitions, bitstreams, and encoding structures and concepts of H.264 / AVC are the same as those in HEVC—therefore, they are described together below.

[0051] Similar to many early video coding standards, H.264 / AVC and HEVC specify the bitstream syntax and semantics, as well as the decoding process for error-free bitstreams. The encoding process is not specified, but the encoder must generate a standard-compliant bitstream. The consistency between the bitstream and the decoder can be verified using a hypothetical reference decoder (HRD). These standards include encoding tools to help handle transmission errors and data loss, but their use in encoding is optional, and no decoding process is specified for error-prone bitstreams.

[0052] The basic unit of input to an H.264 / AVC or HEVC encoder and output to an H.264 / AVC or HEVC decoder is, respectively, the image. The image given as input to the encoder can also be called the source image, and the image decoded by the decoder can be called the decoded image.

[0053] The source image and the decoded image each consist of one or more sample arrays, such as one of the following sets of sample arrays: -Luminance (Y) only (monochrome). - Brightness and two chromaticities (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - Represents an array of other unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0054] Typically, video is encoded in the YUV or YCbCr color space because this has been found to reflect some characteristics of the human visual system and allows for lower-quality representations of the Cb and Cr channels, since human perception is less sensitive to the chromaticity fidelity represented by these channels.

[0055] In H.264 / AVC and HEVC, an image can be a frame or a field. A frame consists of a matrix of luminance samples and possible corresponding chrominance samples. A field is a set of alternating sample rows of a frame and can be used as encoder input when the source signal is interlaced. The chrominance sample array may not exist (therefore monochrome sampling can be used), or the chrominance sample array may be downsampled compared to the luminance sample array. The chrominance format can be summarized as follows: - In monochrome sampling, there is only one sample array, which can be nominally considered as a brightness array. - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luminance array. - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width as the luminance array. - In 4:4:4 sampling, when no separate color plane is used, each of the two chroma arrays has the same height and width as the luminance array.

[0056] In H.264 / AVC and HEVC, the sample array may be encoded into the bitstream as separate color planes, and the individually encoded color planes are decoded from the bitstream accordingly. When using separate color planes, each of them is processed separately (by the encoder and / or decoder) as a picture with monochrome samples.

[0057] Partitioning can be defined as dividing a set into subsets such that each element of the set is located in exactly one of these subsets.

[0058] When describing HEVC encoding and / or decoding operations, the following terminology can be used. A coding block can be defined as an N×N sample block for a given value of N, such that the partitioning from coding tree block to coding block is a partition. A coding tree block (CTB) can be defined as an N×N sample block for a given value of N, such that the partitioning from component to coding tree block is a partition. A coding tree unit (CTU) can be defined as a coding tree block of luma samples, a coding tree block of two chroma samples corresponding to an image with three sample arrays, or a coding tree block of samples of a monochrome image or an image encoded using three separate color planes, along with the syntax structure for encoding the samples. A coding unit (CU) can be defined as a coding block of luma samples, a coding block of two chroma samples corresponding to an image with three sample arrays, or a coding block of samples of a monochrome image or an image encoded using three separate color planes, along with the syntax structure for encoding the samples. A CU with the maximum allowed size can be named an LCU (Maximum Coding Unit) or a coding tree unit (CTU), and video images are divided into non-overlapping LCUs.

[0059] A CU (Currency Unit) consists of one or more prediction units (PUs) and one or more transformation units (TUs). The prediction units (PUs) define the prediction process for samples within the CU, and the transformation units (TUs) define the prediction error coding process for samples within the CU. Typically, a CU consists of square blocks of samples, the size of which can be selected from a predefined set of possible CU sizes. Each PU and TU can be further subdivided into smaller PUs and TUs to correspondingly increase the granularity of the prediction and prediction error coding processes. Each PU has associated prediction information that defines what prediction to apply to the pixels within that PU (e.g., motion vector information for inter-frame prediction PUs and intra-frame prediction directionality information for intra-frame prediction PUs).

[0060] Each TU can be associated with information describing the prediction error decoding process of the samples within that TU (including, for example, DCT coefficient information). For each CU, prediction error coding is typically signaled at the CU level. If there is no prediction error residual associated with that CU, it can be assumed that the CU has no TU. Dividing the image into CUs and into PUs and TUs is typically signaled in the bitstream to allow the decoder to reproduce the intended structure of these units.

[0061] To leverage parallel processing, images can be partitioned into independently coded and independently decodeable image segments (slices or tiles). In HEVC, an image can be partitioned into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, partitioning into tiles forms a regular grid where the height and width of each tile differ from each other by at most one LCU. In HEVC, a slice is defined as an integer number of code tree units contained within an independent slice segment and all subsequent dependent slice segments (if any) preceding the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of code tree units arranged consecutively in tile scan order and contained within a single NAL unit. Each partition from an image to a slice segment is a partition. In HEVC, an independent slice segment is defined as a slice segment whose slice segment header syntax element values ​​are not inferred from the values ​​of previous slice segments, and a dependent slice segment is defined as a slice segment whose slice segment header syntax element values ​​are inferred from the values ​​of previous independent slice segments in the decoding order. In HEVC, a slice header is defined as either a slice header of an independent slice that precedes the current slice, or a slice header of an independent slice preceding the current subordinate slice. The slice header is defined as part of a encoded slice that contains data elements relating to the first or all coded tree units represented in the slice. If tiles are not used, the CUs are scanned within the tiles or the image in the raster scan order of the LCUs. Within an LCU, the CUs have a specific scan order.

[0062] The decoder reconstructs the output video by applying a prediction component similar to that of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, recovering the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it on for display and / or storing it as a prediction reference for subsequent frames in a video sequence.

[0063] Palette-based coding can be used as an alternative or addition to methods that utilize sample value prediction and transform coding to indicate encoded sample values. Palette-based coding refers to a class of methods where a palette, i.e., a set of colors, and associated indices are defined, and the value of each sample within a coding unit is expressed by indicating its index in the palette. Palette-based coding generally achieves good coding efficiency in coding units with relatively few colors (such as image regions representing computer screen content, like text or simple graphics). To improve the coding efficiency of palette coding, various palette index prediction methods can be utilized, or run-length encoding can be applied to the palette indices to efficiently represent larger homogeneous image regions. Furthermore, escape coding can be used when a CU contains sample values ​​that do not repeat within the CU. Escape-coded samples are sent without referencing any arbitrary palette index. Instead, their values ​​are individually indicated for each escape-coded sample.

[0064] Filtering may include, for example, one or more of the following: deblocking filtering, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking filtering, while HEVC includes both deblocking filtering and SAO.

[0065] In a typical video codec, motion information is indicated using motion vectors associated with each motion-compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement between the image block in the picture to be encoded (on the encoder side) or decoded (on the decoder side) and a prediction source block in one of the previously encoded or decoded pictures. To efficiently represent motion vectors, these motion vectors are typically differentially encoded relative to block-specific prediction motion vectors. In a typical video codec, predicted motion vectors are created in a predefined manner, such as by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from co-located blocks in neighboring blocks and / or temporal reference pictures, and to signal the selected candidate as the motion vector predictor. In addition to predicting motion vector values, it is also possible to predict which (or which) reference picture is used for motion-compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously encoded / decoded picture. The reference index is typically predicted from co-located blocks in neighboring blocks and / or temporal reference pictures. Furthermore, typical high-efficiency video codecs employ an additional motion information encoding / decoding mechanism, often referred to as merging / merging mode, where all motion field information (including motion vectors and corresponding reference image indices for each available list of reference images) is predicted and used without any modification / correction. Similarly, motion field information prediction is performed using motion field information from neighboring and / or co-located blocks in the time reference images, and the motion field information used is signaled in a motion field candidate list populated with motion field information from available neighboring / co-located blocks.

[0066] In a typical video codec, the motion-compensated prediction residuals are first transformed using a transform kernel (such as DCT) and then encoded. This is done because some correlation often remains between the residuals, and the transform can help reduce this correlation and provide more efficient coding in many cases.

[0067] Video coding standards and specifications allow encoders to divide encoded images into encoded slices or similar units. Intra-image prediction is typically disabled across slice boundaries. Therefore, a slice can be considered a way to break down an encoded image into independently decodeable fragments. In H.264 / AVC and HEVC, intra-image prediction can be disabled across slice boundaries. Therefore, a slice can be considered a way to break down an encoded image into independently decodeable fragments, and slices are therefore generally considered the basic unit of transmission. In many cases, the encoder can indicate in the bitstream which types of intra-image prediction are disabled across slice boundaries, and the decoder operation takes this information into account, for example, when concluding which prediction sources are available. For example, if adjacent CUs are located in different slices, samples from adjacent CUs can be considered unavailable for intra-frame prediction.

[0068] The basic unit of the output of an H.264 / AVC or HEVC encoder and the input of an H.264 / AVC or HEVC decoder is correspondingly the Network Abstraction Layer (NAL) unit. For transmission over packet-oriented networks or storage in structured files, NAL units can be encapsulated into packets or similar structures. Byte stream formats have been specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide framing structures. The byte stream format separates NAL units from each other by appending a start code to each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm, which adds emulation prevention bytes to the NAL unit payload if the start code appears otherwise. Start code emulation prevention can always be performed to enable direct gateway operation between packet-oriented and stream-oriented systems, regardless of whether the byte stream format is in use. An NAL unit can be defined as a syntax structure containing the following items: an indication of the subsequent data type, and bytes containing the data in RBSP form, interspersed with emulation prevention bytes if necessary. A Raw Byte Sequence Payload (RBSP) can be defined as a syntax structure containing an integer number of bytes encapsulated in NAL units. An RBSP is either empty or has the form of a data bit string containing syntax elements, followed by an RBSP stop bit, and then zero or more subsequent bits equal to 0.

[0069] A NAL element consists of a header and a payload. In H.264 / AVC and HEVC, the NAL element header indicates the type of the NAL element.

[0070] In HEVC, a two-byte NAL cell header is used for all specified NAL cell types. The NAL cell header contains a reserved bit, a six-bit NAL cell type indicator, a three-bit nuh_temporal_id_plus1 indicator for the time layer (which may be required to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered the time identifier of the NAL cell, and the zero-starting TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1 – 1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest time layer. The value of temporal_id_plus1 is required to be non-zero to avoid start code emulation involving two NAL cell header bytes. The bitstream created by excluding all VCL NAL cells with a TemporalId greater than or equal to the selected value and including all other VCL NAL cells remains compliant with the standard. Therefore, an image with a TemporalId equal to tid_value is not used as an inter-frame prediction reference for any image with a TemporalId greater than tid_value. A sublayer, or temporal sublayer, can be defined as a temporally scalable layer (or temporal layer, TL) of a temporally scalable bitstream, consisting of VCL NAL units with specific values ​​for the TemporalId variable and associated non-VCL NAL units. nuh_layer_id can be understood as the scalable layer identifier.

[0071] NAL units can be classified into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically encoded slice NAL units. In HEVC, a VCL NAL unit contains syntax elements representing one or more CUs.

[0072] Non-VCL NAL units can be, for example, one of the following types: sequence parameter set, picture parameter set, supplementary enhancement information (SEI) NAL unit, access unit separator, sequence end NAL unit, bitstream end NAL unit, or padding data NAL unit. Parameter sets may be necessary for reconstructing the decoded picture, while many other non-VCL NAL units are not required for reconstructing the decoded sample values.

[0073] Parameters that remain constant throughout the encoded video sequence can be included in the sequence parameter set. In addition to parameters that may be needed during decoding, the sequence parameter set may optionally include video availability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that can be referenced by one or more picture parameter sets RBSPs or by one or more SEI NAL units containing buffer periodic SEI messages. Picture parameter sets contain such parameters that may remain constant across several encoded pictures. Picture parameter sets RBSPs may include parameters that can be referenced by the coded slice NAL units of one or more encoded pictures.

[0074] In HEVC, the Video Parameter Set (VPS) can be defined as a syntax structure containing syntax elements applicable to zero or more entire encoded video sequences, as determined by the contents of syntax elements found in the SPS referenced by syntax elements found in the PPS referenced by syntax elements found in the header of each slice segment.

[0075] A video parameter set (RBSP) can include parameters that can be referenced by one or more sequence parameter sets (RBSPs).

[0076] The relationship and hierarchy between Video Parameter Set (VPS), Sequence Parameter Set (SPS), and Picture Parameter Set (PPS) can be described as follows. Within the parameter set hierarchy, and in the context of scalable and / or 3D video, the VPS sits one level above the SPS. The VPS may include parameters common to all slices across all (scalable or view) layers throughout the entire encoded video sequence. The SPS includes parameters common to all slices of a specific (scalable or view) layer throughout the entire encoded video sequence and may be shared by multiple (scalable or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalable or view layer within an access unit) and may be shared by all slices in multiple layer representations.

[0077] A VPS can provide information about the dependencies between layers in the bitstream, as well as much other information applicable to all slices of all (scalability or view) layers throughout the entire encoded video sequence. A VPS can be considered to consist of two parts: a base VPS and a VPS extension, where the VPS extension may be present optionally.

[0078] Out-of-band transmission, signaling, or storage may be used additionally or alternatively for purposes other than tolerating transmission errors, such as ease of access or session negotiation. For example, sample entries for tracks in a file conforming to the ISO Basic Media File Format may include parameter sets, while encoded data in the bitstream is stored in another location within that file or in another file. The terms “along the bitstream” (e.g., indicating along the bitstream) or “encoded unit along the bitstream” (e.g., indicating along an encoded tile) may be used in the claims and described embodiments to refer to out-of-band transmission, signaling, or storage in such a way that out-of-band data is correspondingly associated with a bitstream or encoded unit. The terms “decoding along the bitstream” or “decoding an encoded unit along the bitstream” or similar expressions may refer to the out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) correspondingly associated with a bitstream or encoded unit referenced in the decoding.

[0079] The SEI NAL unit can contain one or more SEI messages. These messages are not necessary for decoding the output image, but can assist in related processes such as image output timing, rendering, error detection, error hiding, and resource reservation.

[0080] An encoded image is an encoded representation of an image.

[0081] In HEVC, an encoded picture can be defined as the encoded representation of all the coded tree units containing that picture. In HEVC, an Access Unit (AU) can be defined as a group of NAL units that are associated with each other according to a defined classification rule, are consecutive in the decoding order, and contain at most one picture with an arbitrary nuh_layer_id value. In addition to VCL NAL units containing encoded pictures, Access Units can also contain non-VCL NAL units. The defined classification rule can, for example, associate pictures with the same output time or picture output count value into the same Access Unit.

[0082] A bitstream can be defined as a sequence of bits, in the form of a NAL unit stream or a byte stream, which forms a representation of encoded images and associated data forming one or more encoded video sequences. A first bitstream may follow a second bitstream in the same logical channel, such as within the same file or over the same connection in a communication protocol. A primary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which may be called the End-of-Batch (EOB) NAL unit, and it is the last NAL unit of that bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have a nuh_layer_id equal to 0.

[0083] In H.264 / AVC, an encoded video sequence is defined as a sequence of consecutive access units in decoding order from the IDR access unit (including the IDR access unit) to the next IDR access unit (excluding the IDR access unit) or the end of the bitstream (whichever comes first).

[0084] In HEVC, an encoded video sequence (CVS) can be defined, for example, as a sequence of access units consisting of, in decoding order, IRAP access units with NoRaslOutputFlag equal to 1, followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1, including all subsequent access units up to but not including any subsequent access units (i.e., IRAP access units with NoRaslOutputFlag equal to 1). An IRAP access unit can be defined as an access unit where the base layer picture is an IRAP picture. For each IDR picture, each BLA picture, and each IRAP picture, the value of NoRaslOutputFlag is equal to 1, and the IRAP picture is the first picture in the bitstream of that particular layer in decoding order, and the first IRAP picture following the NAL unit with the same nuh_layer_id value in decoding order. Means may exist for providing the HandleCraAsBlaFlag value to the decoder from an external entity (which can control the decoder, such as a player or receiver). For example, when a player finds a new position in the bitstream or loads a broadcast and begins decoding, starting with a CRA image, it can set HandleCraAsBlaFlag to 1. When HandleCraAsBlaFlag is equal to 1 for a CRA image, the CRA image is processed and decoded as if it were a BLA image.

[0085] In HEVC, an encoded video sequence can be additionally or alternatively (relative to the above specification) designated as the end when a specific NAL unit (which may be referred to as the end-of-sequence (EOS) NAL unit) appears in the bitstream and nuh_layer_id equals 0.

[0086] A Group of Pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures have been decoded. An open GOP is a group of pictures in which pictures preceding the initial intra-picture in the output order may not be correctly decoded when decoding begins from the initial intra-picture of the open GOP. In other words, pictures in an open GOP can (in inter-frame prediction) reference pictures belonging to previous GOPs. The HEVC decoder can identify the intra-pictures that start an open GOP because a specific NAL unit type (i.e., CRA NAL unit type) can be used for its coded slices. A closed GOP is a group of pictures in which all pictures can be correctly decoded when decoding begins from the initial intra-picture of the closed GOP. In other words, no picture in a closed GOP references any picture in a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start from an IDR picture. In HEVC, a closed GOP can also start from a BLA_W_RADL or BLA_N_LP picture. Due to greater flexibility in selecting reference images, open GOP coding structures are potentially more efficient in compression compared to closed GOP coding structures.

[0087] A decoded picture buffer (DPB) can be used in the encoder and / or decoder. Buffered decoded pictures serve two purposes: as a reference in inter-frame prediction and for reordering decoded pictures to the output order. Because H.264 / AVC and HEVC offer significant flexibility for both reference picture marking and output reordering, separate buffers for reference and output pictures could be wasteful of memory resources. Therefore, the DPB can include a unified decoded picture buffering process for both reference and output reordering. When a decoded picture is no longer used as a reference and is no longer needed for output, it can be removed from the DPB.

[0088] In many coding modes of H.264 / AVC and HEVC, reference pictures used for inter-frame prediction are indicated by indices pointing to a list of reference pictures. These indices can be encoded using variable-length coding, which typically results in smaller indices having shorter values ​​for the corresponding syntax elements. In H.264 / AVC and HEVC, two lists of reference pictures (reference picture list 0 and reference picture list 1) are generated for each bidirectional prediction (B) slice, and one list of reference pictures (reference picture list 0) is formed for each inter-frame coded (P) slice.

[0089] Many coding standards (including H.264 / AVC and HEVC) can have a decoding process to derive a reference picture index pointing to a list of reference pictures. This reference picture index can be used to indicate which of a number of reference pictures was used for inter-frame prediction of a particular block. In some inter-frame coding modes, the reference picture index can be encoded into the bitstream by the encoder, or in some other inter-frame coding modes, it can be derived, for example, by the encoder and decoder using adjacent blocks.

[0090] HEVC includes 35 intra-frame prediction modes, including DC mode, planar mode, and 33 angular (direction) prediction modes. DC mode and planar mode are for flat regions (i.e., DC mode represents blocks where pixel values ​​are constant across the entire block) or regions with less structure (i.e., planar mode represents blocks where pixel values ​​gradually change with small planar gradients). Angular modes then provide direction prediction in a very fine-grained manner.

[0091] Motion parameter types or motion information can include, but are not limited to, one or more of the following types: - An indication of the prediction type (e.g., intra-frame prediction, one-way prediction, two-way prediction) and / or the number of reference images; - Indication of prediction direction, such as inter-frame (also known as temporal) prediction, inter-layer prediction, inter-view prediction, view composition prediction (VSP), and inter-component prediction (which may be indicated per reference image and / or per prediction type, and in some embodiments, inter-view prediction and view composition prediction may be considered together as a single prediction direction) and / or - Indication of reference picture type, such as short-term reference picture and / or long-term reference picture and / or interlayer reference picture (which may be indicated, for example, per reference picture). - A reference index and / or any other identifier pointing to a list of reference images (which may be indicated, for example, per reference image, and whose type may depend on the prediction direction and / or reference image type, and may be accompanied by other relevant information segments, such as the list of reference images to which the reference index applies or the like). - Horizontal motion vector components (which may be indicated, for example, per prediction block or per reference index or the like); - Vertical motion vector components (which may be indicated, for example, per prediction block or per reference index or the like); - One or more parameters, such as the image sequence count difference and / or the relative camera spacing between an image containing or associated motion parameters and its reference image, which can be used to scale the horizontal motion vector components and / or the vertical motion vector components during one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference image or per reference index or the like). - The coordinates of the block to which the motion parameters and / or motion information apply, such as the coordinates of the top-left corner sample of the block in units of brightness samples; - The range of blocks to which motion parameters and / or motion information apply (e.g., width and height).

[0092] Compared to previous video coding standards, the H.266 / VVC (Multi-Functional Video Codec) introduces several new encoding tools, such as the following: Intra-frame prediction - 67 intra-frame modes with wide-angle mode extension - 4-tap interpolation filter that depends on block size and mode -Location-dependent intra-frame prediction combination (PDPC) - Cross-component linear model intra-frame prediction (CCLM) - Multi-reference row intra-frame prediction - Intra-frame sub-partitions - Weighted intra-frame prediction with matrix multiplication Inter-frame prediction - Block motion replication with spatial, temporal, historical, and pairwise average merge candidates -Affine motion inter-frame prediction - Sub-block-based temporal motion vector prediction -Adaptive motion vector resolution - 8×8 block-based motion compression for time motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation with an 8-tap interpolation filter for the luminance component and a 4-tap interpolation filter for the chrominance component. - Triangular partition -Intra-frame and inter-frame joint prediction - Merging with MVD (MMVD) -Symmetric MVD encoding - Bidirectional optical flow (BDOF) -Decoder-side motion vector refinement (DMVR) - Bidirectional prediction with CU-level weights Transformation, quantization, and coefficient encoding - Multiple master transform options including DCT2, DST7, and DCT8 - Secondary transformation for low-frequency regions - Sub-block transformation for inter-frame prediction residuals - Dependency quantization increased from maximum QP of 51 to 63. - Transform coefficient encoding with symbolic data hiding -Transform skip residual coding Entropy coding - Arithmetic coding engine with adaptive dual-window probability update -Loop filter - Loop Reshaping - Deblocking filter with strong long filter -Sample adaptive offset - Adaptive Loop Filter Screen content encoding: - Current image reference with reference area limitations 360-degree video encoding -Horizontal orbital motion compensation High-level syntax and parallel processing Reference image management with direct reference image list signaling - A set of tiles with rectangular shapes

[0093] Regarding entropy coding, both the H.265 / HEVC and H.266 / VVC standards use a context-adaptive binary arithmetic encoder (CABAC) to efficiently compress and decompress different syntax elements. As a binary arithmetic encoder, CABAC encodes and decodes syntax elements of 1 bit size. An alternative is to use a non-binary arithmetic encoder, where the encoded and decoded symbols can represent syntax elements with more than one bit of information. The following applies to both binary and non-binary arithmetic encoders, although examples are given for the binary arithmetic encoding case for simplicity.

[0094] In context-adaptive arithmetic coding, each syntax element has a probability estimate for its possible values. These probability estimates are called contexts because they provide a context or model describing the behavior of one or more syntax elements associated with that context. Different sets of contexts (called context categories) can be used, for example, for the luma and chroma components. The list of context categories includes context identifiers (Cids) of those context categories that have been selected for spatial tuning of the state of the arithmetic encoder. In CABAC, a separate list of coding context categories is used for each superblock (such as a coding tree unit (CTU)).

[0095] The efficiency of an arithmetic encoder largely depends on the accuracy of the probability estimates generated for the context. For example, the H.266 standard uses two estimators with different characteristics to estimate the probability that a syntax element is 0 or 1, and uses the average of the two estimators to calculate the activity probability of that context. In this case, the estimators have different adaptive rates; the first reacts faster to changes in the encoded and decoded syntax element values, while the second reacts more slowly, estimating more the long-term trend of the syntax element values. The first estimate can then be considered to have a shorter adaptive window, and the second estimate to have a longer adaptive window.

[0096] Figure 2 This illustrates the image-to-coding tree unit (CTU) or superblock (S0-S). 11 A typical segmentation of ), which serves as the basis for further segmentation into more refined coding units of different shapes, and also provides a framework for some coding tools to initialize or update their state.

[0097] The encoding and decoding of image and video content are typically performed within a nested tree order within a Code Unit (CTU), starting from the top-left CTU (CU) or other coding block and ending at the bottom-right CTU (CU) or other coding block. This is in Figure 3 The image is shown in the diagram, where any CTU in the image is encoded or decoded. The CTU row M contains the CTUs (CTUs). M1 CTU M2 CTU M3 (This has been encoded / decoded.) CTU M2 The nested tree order within a CTU is shown, where numbered (0-6) coding units (CUs) or other coding blocks have been encoded / decoded in numerical order. The CTU in row N is currently being encoded / decoded, where CTU... N1 Encoding / decoding is performed sequentially through a nested tree of numbers (0-8). CTU N2 Encoding / decoding from CTU N2 Start with CU(0) in the top left corner.

[0098] Therefore, when a specific CTU (such as CTU) N1 When encoding or decoding of a CTU is complete, the probability estimate of the arithmetic encoder is tuned to the data present in the lower right region of that CTU. When progressing to a new CTU (e.g., CTU...), N2 When the encoding or decoding process starts again from a new CTU (CTU), the encoding or decoding process starts from a new CTU. N2 The coding unit (CU) or other coding block in the upper left corner continues, where the probability estimate or context state of the arithmetic encoder is not necessarily close to ideal because the processing has jumped to different spatial locations in the image.

[0099] In order to adjust the probabilities of the arithmetic coding engine toward their optimal values ​​in the local neighborhood of the syntax element to be encoded or decoded, some memory is needed to store those bin values ​​or syntax element values ​​in different neighborhoods of the image along with a list of encoding context categories for the corresponding superblock (such as CTU). For example, CTUs from previously encoded CTUs (e.g., CTU...) M2 The bin value or syntax element value of the CU or other coded block (which is spatially located near the new CTU). N2 The top left corner of (e.g., CTU) M2 The coding units 1, 2, 5, and 6 may be more beneficial for adjusting the probabilities of the arithmetic coding engine. It can be readily concluded that the amount of such data may be relatively large and needs to be buffered within the time required for encoding / decoding across multiple CTUs. This amount of data needs to be stored efficiently to minimize the amount of memory required for storage.

[0100] We will now introduce an improved approach using encoding context.

[0101] Based on a coding method in Figure 4 The method is shown in the figure, wherein the method includes: determining (400) a list of context categories; determining (402) a first superblock of an image; determining (404) a first context category from the list of context categories; determining (406) a first bin value belonging to the first context category in the first superblock; using the first bin value to calculate (408) a value for a first probability estimate; encoding a syntax element using the first probability estimate (410); determining (412) a second superblock of an image; determining (414) a second context category from the list of context categories; determining (416) a second bin value belonging to the second context category in the second superblock; using the second bin value to calculate (418) a value for a second probability estimate; and encoding a syntax element using the second probability estimate (420).

[0102] According to another decoding method Figure 5The method is shown in the figure, wherein the method includes: determining (500) a list of context categories; determining (502) a first superblock of an image; determining (504) a first context category from the list of context categories; determining (506) a first bin value belonging to the first context category in the first superblock; using the first bin value to calculate (508) a value for a first probability estimate; decoding the syntax element using the first probability estimate (510); determining (512) a second superblock of an image; determining (514) a second context category from the list of context categories; determining (516) a second bin value belonging to the second context category in the second superblock; using the second bin value to calculate (518) a value for a second probability estimate; and decoding the syntax element using the second probability estimate (520).

[0103] Therefore, in these methods, a subset of arithmetic coding context categories is selected to be included in a context category set, which is allocated for future use to adjust the probabilistic state of the arithmetic encoder with respect to multiple superblocks (such as CTUs). The same context category set can initially be allocated for all CTUs of the image. Using a common but finite-sized list of context categories for multiple CTUs enables more efficient use of memory.

[0104] It should be noted that the term "superblock" is used in this document to refer to the coding unit with the largest allowed size. In H.265 / HEVC and H.266 / VVC, this refers to the LCU (Maximum Coding Unit) or CTU (Coding Tree Unit), but in other video coding standards, another term (such as macroblock) may be used instead. The VP9 video coding standard uses "superblock" to refer to the coding unit with the largest allowed size. In this document, the terms "superblock" and "CTU" are used interchangeably.

[0105] Various embodiments for these methods are described below. It should be noted that the same embodiments apply to both the encoding and decoding processes.

[0106] Therefore, when encoding or decoding the current CTU begins, a position-based update process for the state variables of the arithmetic codec can be triggered. For each CTU or a specific set of CTUs, a selected number of bin values ​​associated with different context categories can be stored for use by the update process.

[0107] According to one embodiment, the number of context categories for storing bin values ​​is limited to N. N can be a predetermined number, such as 96, 128, 192, or 256, or a number derived using a given set of rules. Alternatively, N can be indicated in the video or image bitstream.

[0108] The list of context categories (i.e., the list of context categories already included in the set of context categories for which bin values ​​are stored) is constructed and maintained by the arithmetic encoder. Each context category in this list is also associated with a stored bin value for that context category. Bin values ​​can be stored in different ways. For example, there can be an array of bin values ​​and a counter indicating how many bin values ​​are currently included for a given context category. Each context category can be represented by its unique identifier or context ID in the list of context categories. Such an identifier can be, for example, an integer.

[0109] The context class for storing bin values ​​can be determined by selecting the top N context classes that satisfy a given condition, according to the order in which the syntax elements and their associated context classes appear in the bitstream. According to one embodiment, the bottom coding unit (CU) of the CTU is included as a candidate CU for this process. Different selections may be made in alternative embodiments.

[0110] According to one embodiment, the method further includes updating the context category list at the beginning of an image, the beginning of a slice, or the beginning of a superblock line.

[0111] Therefore, the same list of context categories can be used for the entire image, per slice, or per superblock line. The list of context categories can be partially updated (where a subset of context categories is removed from the list) or entirely updated (where all context categories previously included in the list are removed). An alternative to resetting the list is to initialize it using the context category list of another image or slice. Resetting typically also includes clearing all stored bin values ​​and any associated counters for different context categories. If the given conditions for including a new context category are met, the new context category can be included in the context category list as it appears in the bitstream.

[0112] According to one embodiment, the list of context categories has a defined maximum size.

[0113] By limiting the size of the context category list, efficient memory usage is facilitated. Furthermore, a condition for including a new context category is that the defined maximum size of the context category list cannot be exceeded.

[0114] According to one embodiment, the method further includes constructing a list of context categories by including context categories from at least two superblocks.

[0115] The selected set of context categories can then include context categories from multiple coding tree units.

[0116] According to one embodiment, the method further includes constructing a list of context categories by including at most a determined maximum number of context categories from a single superblock.

[0117] Therefore, by setting the maximum number of context categories that can be included from a single encoding tree unit, it becomes possible to generalize the selected set of context categories, since the set cannot then be populated with only syntax elements from a single encoding tree unit.

[0118] Some implementations can be illustrated by the following pseudocode example, where N is the maximum number of context categories whose bin values ​​can be stored, and M is the maximum number of bin values ​​that can be stored for each such context category: numberOfActiveContextClasses = 0 contextClassStorage = empty()for a CTU in picture: for a CU located on the bottom of the CTU:for a bin used to represent syntax elements of the CU:contextClass = contextClassOf(bin) if contextClass is already incontextClassStorage: if numberOfBinsStored for contextClass <M: store bin to contextClassStorage for contextClasselse if numberOfActiveContextClasses<N: include contextClassin contextClassStorage increase numberOfActiveContextClassesby 1 store bin to contextClassStorage for contextClass

[0119] According to one embodiment, the method further includes constructing a context category list having a determined number of context categories calculated using an index of the encoding unit.

[0120] As an alternative example, the number of context categories that can be included in the context category list can be determined as a function of the CTU index. This allows for a more progressive construction of the context category list and potentially allows for a more representative selection of context categories within the list. For example, the maximum number of context categories for a given CTU can be determined as X. The minimum values ​​of CTUind and N, where X is a parameter controlling the rate at which new context categories are included in the list, CTUind is the index of the CTU in the encoding order, and N is the total maximum size of the context category list. In this case, the pseudocode example can be restated as: numberOfActiveContextClasses = 0 contextClassStorage = empty()for a CTU in picture: maxNumContexts = min(X CTUind, N) for a CUlocated on the bottom of the CTU: for a bin used to represent syntaxelements of the CU: contextClass = contextClassOf(bin)if contextClass is already in contextClassStorage: ifnumberOfBinsStored for contextClass <M: store bin tocontextClassStorage for contextClass else ifnumberOfActiveContextClasses<maxNumContexts: includecontextClass in contextClassStorage increasenumberOfActiveContextClasses by 1 store bin tocontextClassStorage for contextClass

[0121] As another example, the maximum number of context categories that can be included for a specific CTU can be limited to a certain value. In this case, the maximum number of context categories that cannot be exceeded when processing a given CTU can be determined as the minimum of the number of context categories at the start of CTU processing plus Y and N, where Y is a parameter controlling how many new context categories can be included for the CTU, and N is the total maximum size of the list of context categories. In this case, the pseudo-example can be restated as: numberOfActiveContextClasses = 0 contextClassStorage = empty()for a CTU in picture: maxNumContexts = min(numberOfActiveContextClasses+ Y, N) for a CU located on the bottom of the CTU: for a binused to represent syntax elements of the CU: contextClass =contextClassOf(bin) if contextClass is already incontextClassStorage: if numberOfBinsStored for contextClass <M: store bin to contextClassStorage for contextClasselse if numberOfActiveContextClasses<maxNumContexts: includecontextClass in contextClassStorage increasenumberOfActiveContextClasses by 1 store bin tocontextClassStorage for contextClass

[0122] According to one embodiment, the method includes storing bin values ​​associated with context categories included in the list of context categories in a superblock-specific buffer.

[0123] In the above embodiments, a single list of context categories per image or per slice is sufficient, enabling the sharing of the context category list among multiple CTUs. However, each CTU can further use its own bin value storage buffer to store bin values ​​associated with the context categories included in the context category list. As an example, the storage buffer for bin values ​​may include two elements: one for the stored bin values ​​and a counter indicating how many bin values ​​associated with a given context category have been stored. Alternatively, two-element arrays may exist. One array contains the stored bin values, and the other array contains a counter indicating the number of stored bin values ​​for each context category. The context category list may be implemented, for example, as a list, vector, array, or other kind of data structure. The context category list may include, for example, an integer or a context identifier that uniquely identifies a context category in the set of all context categories.

[0124] Applying a shared list of context categories to multiple CTUs minimizes the need to store context identifiers or context IDs, and the number of context categories used in encoding or decoding is less than the number of bin value storage lists. This configuration allows multiple CTUs to use the same list of context categories to determine which bin values ​​are stored or used to update the probability estimates of the arithmetic encoder.

[0125] Updates to the state variables of an arithmetic encoder can be performed in different ways using stored bin values. For example, it can be done based on discontinuities in the scanning of coding units within an image, where bin values ​​or syntax element values ​​can be stored in memory for updating the probabilistic state of subsequent coding units or coding tree units. As another example, the state of the arithmetic encoder can be updated based on buffered context index values ​​and associated bin values, where the context indices to be buffered are determined by selecting a number of context index values ​​and associated bin values ​​in the order in which they appear in the source coding tree units.

[0126] According to one embodiment, the method includes: updating at least a first probability estimate by determining a first context category using a list of context categories and bin values ​​stored for a first superblock; and updating at least a second probability estimate by determining a second context category using the list of context categories and bin values ​​stored for a second superblock.

[0127] According to one embodiment, the method includes: updating at least one probability estimate associated with a first context category in a first superblock based on stored bin values ​​of a second superblock; and updating at least one probability estimate associated with a second context category in a third superblock based on stored bin values ​​of a fourth superblock, wherein the first and second context categories are determined using the same list of context categories.

[0128] The above embodiments are examples of the benefits of sharing the same list of context categories among multiple CTUs (superblocks), where the probability estimate associated with a certain context category in a certain CTU can be updated based on the bin values ​​stored in another (neighboring) CTU.

[0129] According to one embodiment, the method includes sharing two lists of context categories among multiple superblocks, wherein a first list of context categories is used to store context categories for which bin values ​​are stored to update probability estimates for the next superblock row, and a second list of context categories is used to identify which context categories are stored on the previous superblock row and to update probability estimates for the current superblock row.

[0130] Therefore, the alternative embodiments operate based on CTU (i.e., superblock) rows. In this document, two context category lists are maintained: a first context category list is used to collect context categories for the current CTU row, and a second context category list represents the final context category list for previous CTU rows (i.e., CTU rows above the current CTU row). When a context update procedure is invoked for a CTU before encoding or decoding begins, the second context category list is used to determine which contexts to update using the stored bin values ​​of each CTU above the current CTU. The first context category list is used when storing the selected bin values ​​for the current CTU. This allows for the selection of different context categories for each CTU row and can provide better encoding efficiency in some scenarios, at the cost of maintaining two context category lists instead of one.

[0131] According to one embodiment, the method includes resetting a first context category list before processing a first superblock in a superblock row; and skipping the reset for a second context category list.

[0132] With this arrangement, the first context category list and the second context category list can be swapped at the end of the CTU line because the context category list used to determine which contexts to update for the current CTU line is no longer needed. It can then be cleared and used as the context category list for selecting new context categories for the bin value buffer on the next CTU line.

[0133] Some of the above embodiments are through Figure 6 The example illustrates this, depicting an example memory array layout. ContextList represents a possible way to implement a list of context categories, shared among multiple CTUs, preferably for the entire image. The context category list includes context identifiers (Cids) for those context categories selected for spatial tuning of the arithmetic encoder states. In this example, there are as many BinStorage tables as there are CTU columns. It should be noted that one BinStorage table per CTU column is usually sufficient, as the same table can first be used to update the probability estimate of the arithmetic encoder at the beginning of processing a CTU, and then be cleared and used to store the bin values ​​associated with that CTU as processing progresses.

[0134] For illustrative purposes, in this example, the same BinStorage table is used to store bin values ​​(Bins) and a counter (C) indicating how many bin values ​​have been stored for a CTU. However, it should be noted that in a real implementation, it is not necessary to store the context identifier in the BinStorage table, as the slot number can be used to determine which context categories in the ContextList table are associated with the bin values ​​stored in each group in the BinStorage table. Furthermore, it is not necessary to store the slot numbers shown in the different tables, as slot numbers are typically available as memory offsets starting from each array.

[0135] An apparatus according to one aspect includes: components for determining a list of context categories; components for determining a first superblock of an image; components for determining a first context category from the list of context categories; components for determining a first bin value belonging to the first context category in the first superblock; components for using the first bin value to compute a value for a first probability estimate; components for encoding syntax elements using the first probability estimate; components for determining a second superblock of an image; components for determining a second context category from the list of context categories; components for determining a second bin value belonging to the second context category in the second superblock; components for using the second bin value to compute a value for a second probability estimate; and components for encoding syntax elements using the second probability estimate.

[0136] An apparatus according to one aspect includes: components for determining a list of context categories; components for determining a first superblock of an image; components for determining a first context category from the list of context categories; components for determining a first bin value belonging to the first context category in the first superblock; components for calculating a value for a first probability estimate using the first bin value; components for decoding a syntax element using the first probability estimate; components for determining a second superblock of an image; components for determining a second context category from the list of context categories; components for determining a second bin value belonging to the second context category in the second superblock; components for calculating a value for a second probability estimate using the second bin value; and components for decoding a syntax element using the second probability estimate.

[0137] According to one embodiment, the apparatus includes components for updating a list of context categories at the beginning of an image, at the beginning of a slice, or at the beginning of a superblock line.

[0138] According to one embodiment, the list of context categories has a defined maximum size.

[0139] According to one embodiment, the apparatus includes a component for constructing a list of context categories by including context categories from at least two superblocks.

[0140] According to one embodiment, the apparatus includes a component for constructing a list of context categories by including at most a determined maximum number of context categories from a single superblock.

[0141] According to one embodiment, the apparatus includes components for constructing a list of context categories having a determined number of context categories calculated using an index of an encoding unit.

[0142] According to one embodiment, the apparatus includes a component for storing bin values ​​associated with context categories included in a list of context categories in a superblock-specific buffer.

[0143] According to one embodiment, the apparatus includes: components for updating at least a first probability estimate by determining a first context category using a list of context categories and bin values ​​stored for a first superblock; and components for updating at least a second probability estimate by determining a second context category using a list of context categories and bin values ​​stored for a second superblock.

[0144] According to one embodiment, the apparatus includes: a component for updating at least one probability estimate associated with a first context category in a first superblock based on stored bin values ​​of a second superblock; and a component for updating at least one probability estimate associated with a second context category in a third superblock based on stored bin values ​​of a fourth superblock, wherein the first context category and the second context category are determined using the same list of context categories.

[0145] According to one embodiment, the apparatus includes components for sharing two lists of context categories among a plurality of superblocks, wherein a first list of context categories is used to store context categories, bin values ​​for context categories are stored for updating probability estimates for the next superblock row, and a second list of context categories is used to identify which context categories are stored on the previous superblock row and is used for updating probability estimates for the current superblock row.

[0146] According to one embodiment, the apparatus includes: a component for resetting a first context category list before processing a first superblock in a superblock row; and a component for skipping the reset of a second context category list.

[0147] According to one embodiment, a superblock is a coding tree unit (CTU), a maximum coding unit (LCU), or a macroblock.

[0148] In another aspect, an apparatus is provided, comprising: at least one processor and at least one memory, wherein code is stored on the at least one memory, the code, when executed by the at least one processor, causes the apparatus to perform at least: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; using the first bin value to compute a value for a first probability estimate; encoding a syntax element using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; using the second bin value to compute a value for a second probability estimate; and encoding a syntax element using the second probability estimate.

[0149] In another aspect, an apparatus is provided, comprising: at least one processor and at least one memory, wherein code is stored on the at least one memory, the code, when executed by the at least one processor, causes the apparatus to at least perform: determining a list of context categories; determining a first superblock of an image; determining a first context category from the list of context categories; determining a first bin value belonging to the first context category in the first superblock; using the first bin value to compute a value for a first probability estimate; decoding a syntax element using the first probability estimate; determining a second superblock of an image; determining a second context category from the list of context categories; determining a second bin value belonging to the second context category in the second superblock; using the second bin value to compute a value for a second probability estimate; and decoding a syntax element using the second probability estimate.

[0150] According to one embodiment, the apparatus includes code configured to update the context category list at the beginning of an image, the beginning of a slice, or the beginning of a superblock line.

[0151] According to one embodiment, the list of context categories has a defined maximum size.

[0152] According to one embodiment, the apparatus includes code configured to construct a list of context categories by including context categories from at least two superblocks.

[0153] According to one embodiment, the apparatus includes code configured to construct a list of context categories by including at most a determined maximum number of context categories from a single superblock.

[0154] According to one embodiment, the apparatus includes code configured to cause the apparatus to construct a list of context categories having a determined number of context categories calculated using an index of an encoding unit.

[0155] According to one embodiment, the apparatus includes code configured to store bin values ​​associated with context categories included in a list of context categories in a superblock-specific buffer.

[0156] According to one embodiment, the apparatus includes: code configured to update at least a first probability estimate by determining a first context category using a list of context categories and bin values ​​stored for a first superblock; and code configured to update at least a second probability estimate by determining a second context category using a list of context categories and bin values ​​stored for a second superblock.

[0157] According to one embodiment, the apparatus includes: code configured to update at least one probability estimate associated with a first context category in a first superblock based on stored bin values ​​of a second superblock; and code configured to update at least one probability estimate associated with a second context category in a third superblock based on stored bin values ​​of a fourth superblock, wherein the first context category and the second context category are determined using the same list of context categories.

[0158] According to one embodiment, the apparatus includes code configured to share two lists of context categories among a plurality of superblocks, wherein a first list of context categories is used to store context categories, bin values ​​for context categories are stored for updating probability estimates for the next superblock line, and a second list of context categories is used to identify which context categories are stored on the previous superblock line and is used for updating probability estimates for the current superblock line.

[0159] According to one embodiment, the apparatus includes: code configured to reset a first context category list before processing a first superblock in a superblock line; and code configured to skip the reset of a second context category list.

[0160] According to one embodiment, a superblock is a coding tree unit (CTU), a maximum coding unit (LCU), or a macroblock.

[0161] Such devices may include, for example, the attached Figure 1a , Figure 1b as well as Figures 7 to 10 All or a subset of the functional units disclosed in any one of them are used to implement these embodiments.

[0162] Such devices also include code stored in the at least one memory, which, when executed by the at least one processor, causes the device to perform one or more embodiments disclosed herein.

[0163] Figure 7 A schematic block diagram of an exemplary device or electronic device 50 is shown, which may be combined with a codec according to embodiments of the present invention. Figure 8 The layout of the apparatus according to an example embodiment is shown.

[0164] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be understood that embodiments of the present invention can be implemented in any electronic device or apparatus that may require encoding and decoding or encoding or decoding of video images.

[0165] Device 50 may include a housing 30 for accommodating and protecting the device. Device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the invention, the display may be any suitable display technology suitable for displaying images or videos. Device 50 may also include a keypad 34. In other embodiments of the invention, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or a data input system as part of a touch-sensitive display.

[0166] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may also include an audio output device, which in embodiments of the invention may be any of the following: a handset 38, a speaker, or an analog or digital audio output connection. The device 50 may also include a battery (or, in other embodiments of the invention, may be powered by any suitable mobile energy device, such as a solar cell, fuel cell, or clockwork generator). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as Bluetooth wireless connectivity or USB / FireWire wired connectivity.

[0167] The device 50 may include a controller 56, a processor, or a processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which, in embodiments of the invention, may store data in the form of image and audio data, and / or instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54, which is adapted to perform encoding and decoding of audio and / or video data, or to assist in encoding and decoding performed by the controller.

[0168] The device 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC card reader, for providing user information and suitable for providing authentication information to authenticate and authorize the user at the network.

[0169] The device 50 may include a wireless interface circuitry 52 connected to the controller, the wireless interface circuitry 52 being adapted to generate wireless communication signals, for example, for communicating with a cellular communication network, a wireless communication system, or a wireless local area network. The device 50 may also include an antenna 44 connected to the wireless interface circuitry 52 for transmitting radio frequency signals generated at the wireless interface circuitry 52 to other devices and for receiving radio frequency signals from other devices.

[0170] Apparatus 50 may include a camera capable of recording or detecting individual frames, which are then passed to codec 54 or a controller for processing. The apparatus may receive video image data for processing from another device before transmission and / or storage. Apparatus 50 may also receive images for encoding / decoding via wireless or wired connection. The structural elements of apparatus 50 described above represent examples of components used to perform the corresponding functions.

[0171] about Figure 9 This illustrates an example of a system in which embodiments of the present invention can be utilized. System 10 includes a plurality of communication devices that can communicate over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet LANs, token ring LANs, wide area networks, and the Internet.

[0172] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the present invention.

[0173] For example, Figure 9 The system shown illustrates a representation of mobile phone network 11 and Internet 28. The connection to Internet 28 can include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, power lines, and similar communication paths.

[0174] The example communication devices shown in System 10 may include, but are not limited to, electronic devices or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a laptop computer 22. When carried by a mobile individual, apparatus 50 may be stationary or mobile. Apparatus 50 may also be located in a means of transport, including but not limited to cars, trucks, taxis, buses, trains, ships, airplanes, bicycles, motorcycles, or any similar suitable mode of transport.

[0175] These embodiments can also be implemented in set-top boxes; i.e., digital TV receivers, which may or may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PCs) with hardware or software or a combination thereof that implement encoders / decoders, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems that provide hardware / software-based encoding.

[0176] Some or other devices can send and receive calls and messages, and communicate with service providers via a wireless connection 25 to base station 24. Base station 24 can connect to network server 26, which allows communication between mobile phone network 11 and Internet 28. The system may include additional communication equipment and various types of communication devices.

[0177] These communication devices can communicate using various transmission technologies, including but not limited to Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol-Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), Email, Instant Messaging Service (IMS), Bluetooth, IEEE 802.11, and any similar wireless communication technologies. The communication devices involved in implementing the various embodiments of this invention can communicate using various media, including but not limited to wireless, infrared, laser, cable connections, and any suitable connection.

[0178] Figure 10 This is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. Data source 1510 provides a source signal in analog, uncompressed digital, or compressed digital formats, or any combination of these formats. Encoder 1520 may include or be connected to preprocessing, such as data format conversion and / or filtering of the source signal. Encoder 1520 encodes the source signal into an encoded media bitstream. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located within virtually any type of network. Alternatively, the bitstream can be received from local hardware or software. Encoder 1520 is capable of encoding more than one media type, such as audio and video, or may require more than one encoder 1520 to encode different media types of the source signal. Encoder 1520 may also receive synthetically generated input, such as graphics and text, or be capable of generating an encoded bitstream of synthetic media. In the following description, for simplicity, only the processing of one encoded media bitstream of one media type is considered. However, it should be noted that a typical live broadcast service includes several streams (typically at least one audio, video, and text caption stream). It should also be noted that the system may include many encoders, but only one encoder 1520 is shown in the figure for simplicity without loss of generality. It should also be understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles apply to the corresponding decoding process, and vice versa.

[0179] The encoded media bitstream can be transferred to memory 1530. Memory 1530 can include any type of mass storage to store the encoded media bitstream. The format of the encoded media bitstream in memory 1530 can be a basic self-contained bitstream format, or one or more encoded media bitstreams can be encapsulated into a container file, or the encoded media bitstream can be encapsulated into a fragment format suitable for DASH (or a similar streaming system) and stored as a sequence of fragments. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown) can be used to store the one or more media bitstreams in the file and create file format metadata, which can also be stored in the file. Encoder 1520 or memory 1530 can include the file generator, or the file generator is operatively attached to encoder 1520 or memory 1530. Some systems operate "in real time," omitting the memory and transmitting the encoded media bitstream directly from encoder 1520 to transmitter 1540. The encoded media bitstream can then be transmitted to transmitter 1540, also known as a server, as needed. The format used in transmission can be a basic self-contained bitstream format, a packet stream format, a fragment format suitable for DASH (or a similar streaming system), or one or more encoded media bitstreams can be encapsulated into a container file. Encoder 1520, memory 1530, and server 1540 can reside in the same physical device, or they can be included in separate devices. Encoder 1520 and server 1540 can operate with live streaming content, in which case the encoded media bitstream is typically not permanently stored, but rather buffered briefly in content encoder 1520 and / or server 1540 to smooth out variations in latency, transmission delay, and encoded media bitrate.

[0180] Server 1540 uses a communication protocol stack to send the encoded media bitstream. This stack can include, but is not limited to, one or more of Real-Time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). When the communication protocol stack is packet-oriented, server 1540 encapsulates the encoded media bitstream into packets. For example, when using RTP, server 1540 encapsulates the encoded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system can contain more than one server 1540, but for simplicity, the following description considers only one server 1540.

[0181] If media content is encapsulated in a container file for use in memory 1530 or for inputting data to transmitter 1540, transmitter 1540 may include a "transmission file parser" (not shown) or be operatively attached to it. Specifically, if the container file itself is not transmitted, but at least one of the encoded media bitstreams it contains is encapsulated for transmission over a communication protocol, the transmission file parser locates the appropriate portion of the encoded media bitstream to be transmitted over that communication protocol. The transmission file parser may also help create the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions, such as prompt tracks in ISOBMFF, for encapsulating at least one of the contained media bitstreams over the communication protocol.

[0182] Server 1540 may or may not be connected to gateway 1550 via a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. The gateway may also be, or alternatively, referred to as a middlebox. For DASH, the gateway may be an edge server (of a CDN) or a web proxy. Note that a system can typically include any number of gateways or similar devices, but for simplicity, the following description considers only one gateway 1550. Gateway 1550 can perform different types of functions, such as packet stream conversion from one communication protocol stack to another, data stream merging and forking, and data stream manipulation based on downlink and / or receiver capabilities, such as controlling the bit rate of the forwarded stream according to general downlink network conditions. In various embodiments, gateway 1550 may be a server entity.

[0183] The system includes one or more receivers 1560, typically capable of receiving, demodulating, and decapsulating transmitted signals as encoded media bitstreams. The encoded media bitstream can be transferred to a recording memory 1570. The recording memory 1570 can include any type of mass storage to store the encoded media bitstream. The recording memory 1570 may optionally or additionally include computational memory, such as random access memory. The format of the encoded media bitstream in the recording memory 1570 can be a basic self-contained bitstream format, or one or more encoded media bitstreams can be encapsulated into a container file. If multiple associated encoded media bitstreams exist, such as audio and video streams, a container file is typically used, and the receiver 1560 includes or is attached to a container file generator that produces the container file from the input stream. Some systems operate "in real-time," omitting the recording memory 1570 and transferring the encoded media bitstream directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recording stream (e.g., the most recent 10-minute extract of the recording stream) is kept in the recording memory 1570, while any earlier recorded data is discarded from the recording memory 1570.

[0184] The encoded media bitstream can be transferred from the recording memory 1570 to the decoder 1580. If multiple associated encoded media bitstreams, such as audio and video streams, exist and are encapsulated in a container file, or if a single media bitstream is encapsulated in a container file for ease of access, a file parser (not shown) is used to decapsulate each encoded media bitstream from the container file. The recording memory 1570 or the decoder 1580 may include the file parser, or the file parser may be attached to either the recording memory 1570 or the decoder 1580. It should also be noted that the system may include many decoders, but only one decoder 1580 is discussed here for simplicity without loss of generality.

[0185] The encoded media bitstream can be further processed by decoder 1580, the output of which is one or more uncompressed media streams. Finally, renderer 1590 can reproduce the uncompressed media stream, for example, using a speaker or display. Receiver 1560, recording memory 1570, decoder 1580, and renderer 1590 can reside in the same physical device, or they can be included in separate devices.

[0186] Transmitter 1540 and / or gateway 1550 can be configured to perform switching between different representations, such as switching between different viewports for 360-degree video content, view switching, bitrate adaptation, and / or fast startup, and / or transmitter 1540 and / or gateway 1550 can be configured to select the representation(s) to be transmitted. Switching between different representations can occur for various reasons, such as in response to a request from receiver 1560 or general conditions, such as the throughput of the network transmitting the bitstream. In other words, receiver 1560 can initiate a switching between representations. Requests from the receiver can be, for example, requests for segments or sub-segments of a different representation than before, requests for changes to the transmitted scalable layers and / or sub-layers, or changes to a rendering device with different capabilities compared to the previous device. Requests for segments can be HTTP GET requests. Requests for sub-segments can be HTTP GET requests with byte ranges. Additionally or alternatively, bitrate adjustment or bitrate adaptation can be used, for example, to provide so-called fast start in streaming services, where the bitrate of the transmitted stream is lower than the channel bitrate after the start of or random access streaming, so as to start playback immediately and achieve a buffer occupancy level that can tolerate occasional packet delays and / or retransmissions. Bitrate adaptation may include multiple representation or layer up switching operations and representation or layer down switching operations occurring in various sequences.

[0187] Decoder 1580 can be configured to perform switching between different representations, such as switching between different viewports for 360-degree video content, view switching, bitrate adaptation, and / or fast startup, and / or decoder 1580 can be configured to select the transmitted representation(s). Switching between different representations can occur for various reasons, such as to achieve faster decoding operations or to adapt the transmitted bitstream (e.g., in terms of bitrate) to the general conditions of the network transmitting the bitstream, such as throughput. For example, if the device including decoder 1580 is multitasking and devoting computational resources to purposes other than decoding the video bitstream, faster decoding operations may be needed. In another example, faster decoding operations may be needed when content is played back at a faster speed than normal playback speed, such as two or three times faster than the usual real-time playback rate.

[0188] In the foregoing, some embodiments have been described with reference to and / or using the terminology of HEVC and / or VVC. It should be understood that the embodiments can be similarly implemented using any video encoder and / or video decoder.

[0189] In the foregoing description of example embodiments with reference to the encoder, it should be understood that the resulting bitstream and decoder may contain corresponding elements. Similarly, in the description of example embodiments with reference to the decoder, it should be understood that the encoder may have a structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0190] The embodiments of the invention described above are presented as a codec with separate encoder and decoder devices to aid in understanding the processes involved. However, it should be understood that the device, structure, and operation can be implemented as a single encoder-decoder device / structure / operation. Furthermore, it is possible for the encoder and decoder to share some or all of their common elements.

[0191] While the foregoing examples describe embodiments of the invention operating within a codec in an electronic device, it should be understood that the invention as defined in the claims can be implemented as part of any video codec. Thus, for example, embodiments of the invention can be implemented in a video codec that enables video encoding via a fixed or wired communication path.

[0192] Therefore, user equipment may include video codecs such as those described above in the embodiments of the present invention. It should be understood that the term user equipment is intended to cover any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, or portable web browsers.

[0193] Furthermore, the components of a Public Land Mobile Network (PLMN) may also include the video codecs described above.

[0194] Generally, various embodiments of the present invention can be implemented in hardware or dedicated circuit systems, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although the invention is not limited thereto. While various aspects of the invention may be illustrated and described as block diagrams, flowcharts, or other graphical representations, it is fully understood that the blocks, devices, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuit systems or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0195] Embodiments of the present invention can be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. It should also be noted in this regard that any block of the logical flow in the figures can represent a program step, or interconnected logic circuits, blocks, and functions, or a combination of program steps and logic circuits, blocks, and functions. The software can be stored on physical media, such as memory chips, or memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, and CDs.

[0196] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor can be of any type suitable for the local technical environment and, as a non-limiting example, can include one or more of the following: general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures.

[0197] Embodiments of the present invention can be implemented in various components such as integrated circuit modules. The design of integrated circuits is largely a highly automated process. Complex and powerful software tools can be used to transform logic-level designs into semiconductor circuit designs ready to be etched and formed on semiconductor substrates.

[0198] Programs, such as those provided by Synopsys in Mountain View, California, and Cadence Design in San Jose, California, use sophisticated design rules and pre-stored libraries of design modules to automatically place conductors and position components on semiconductor chips. Once the semiconductor circuit design is complete, the resulting design, in a standardized electronic format (such as Opus, GDSII, etc.), can be transferred to a semiconductor manufacturing facility or "fab" for fabrication.

[0199] The foregoing description has provided a complete and informative description of exemplary embodiments of the invention through exemplary and non-limiting examples. However, various modifications and adjustments may become apparent to those skilled in the art when read in conjunction with the accompanying drawings and appended claims, given the foregoing description. Nevertheless, all such and similar modifications taught in this invention will still fall within the scope of this invention.

Claims

1. An apparatus comprising: A component used to determine the list of context categories; The component used to determine the first superblock of the image; The component used to determine the first context category from the list of context categories: Components used to determine the first bin value of the first context category belonging to the first superblock; A component for calculating a value for a first probability estimate using the first bin value; A component for encoding syntax elements using the first probability estimate; Components used to determine the second superblock of the image; A component used to determine a second context category from the list of context categories; Components used to determine the second bin value belonging to the second context category in the second superblock; A component used to calculate a value for a second probability estimate using the second bin value; as well as A component used to encode syntax elements using the second probability estimate.

2. An apparatus comprising: A component used to determine the list of context categories; The component used to determine the first superblock of the image; A component used to determine a first context category from the list of context categories; Components used to determine the first bin value of the first context category belonging to the first superblock; A component for calculating a value for a first probability estimate using the first bin value; A component for decoding syntax elements using the first probability estimate; Components used to determine the second superblock of the image; A component used to determine a second context category from the list of context categories; Components used to determine the second bin value belonging to the second context category in the second superblock; A component used to calculate a value for a second probability estimate using the second bin value; as well as A component for decoding syntax elements using the second probability estimate.

3. The apparatus according to claim 1 or 2, comprising: A component used to update the list of context categories at the beginning of the image, the beginning of a slice, or the beginning of a superblock line.

4. The apparatus according to any of the preceding claims, wherein the context category list has a defined maximum size.

5. The apparatus according to any of the preceding claims, comprising: A component used to construct the list of context categories by including context categories from at least two superblocks.

6. The apparatus according to any of the preceding claims, comprising: A component for constructing the list of context categories by including at most a determined maximum number of context categories from a single superblock.

7. The apparatus according to any of the preceding claims, comprising: The component for constructing the context category list has a determined number of context categories calculated using the index of the encoding unit.

8. The apparatus according to any of the preceding claims, comprising: A component for storing the bin value associated with the context category included in the list of context categories in a superblock-specific buffer.

9. The apparatus according to any of the preceding claims, comprising: The component used to update at least the first probability estimate by using the context category list and the bin value stored for the first superblock to determine the first context category; as well as The component used to update at least the second probability estimate by using the context category list and the bin value stored for the second superblock to determine the second context category.

10. The apparatus according to any of the preceding claims, comprising: A component for updating at least one probability estimate related to the first context category in the first superblock based on the stored bin value of the second superblock; as well as A component for updating at least one probability estimate associated with the second context category in the third superblock based on the stored bin values ​​of the fourth superblock, wherein the first context category and the second context category are determined using the same list of context categories.

11. The apparatus according to claim 1 or 2, comprising: A component used to share two lists of context categories across multiple superblocks, where A first list of context categories is used to store context categories, and bin values ​​for those context categories are stored to update the probability estimate for the next superblock row. The second list of context categories is used to identify which context categories were stored in the previous superblock line and is used to update the probability estimate for the current superblock line.

12. The apparatus of claim 11, comprising: A component for resetting the first context category list before processing the first superblock in the superblock row; as well as The component used to skip the reset for the second context category list.

13. The apparatus according to any of the preceding claims, wherein the superblock is a coding tree unit (CTU), a maximum coding unit (LCU), or a macroblock.

14. A method comprising: Determine the list of context categories; Determine the first superblock of the image; Determine a first context category from the list of context categories; Determine the first bin value belonging to the first context category in the first superblock; The first bin value is used to calculate the value for the first probability estimate; The syntax elements are encoded using the first probability estimate; Determine the second superblock of the image; Determine a second context category from the list of context categories; Determine the second bin value belonging to the second context category in the second superblock; Use the second bin value to calculate the value for the second probability estimate; as well as The syntax elements are encoded using the second probability estimate.

15. A method comprising: Determine the list of context categories; Determine the first superblock of the image; Determine a first context category from the list of context categories; Determine the first bin value belonging to the first context category in the first superblock; The first bin value is used to calculate the value for the first probability estimate; The syntax elements are decoded using the first probability estimate; Determine the second superblock of the image; Determine a second context category from the list of context categories; Determine the second bin value belonging to the second context category in the second superblock; Use the second bin value to calculate the value for the second probability estimate; as well as The syntax elements are decoded using the second probability estimate.

Images