Method and apparatus for block-level window size update for arithmetic coding - Patents.com

Abstract

This disclosure relates to video coding with block-level window size updates for arithmetic coding. The probability update rate may be a function of a count of occurrences for an associated syntax / context when parsing a bitstream. It may also be a function of the number of distinct symbol values ​​for the associated syntax / context. The count value is restarted with a predefined value at the block level. The probability or cumulative distribution function (CDF) update rate for a particular syntax / context may be reinitialized with a predefined value at the block level.

Description

[Technical field]

[0001] Priority This application claims priority to U.S. Application No. 18 / 060,259, entitled "Block-Level Window Size Update for Arithmetic Coding," filed November 30, 2022, which claims the benefit of priority to U.S. Provisional Application No. 63 / 306,849, entitled "Block-Level Window Size Update for Arithmetic Coding," filed February 4, 2022, each of which is incorporated by reference in its entirety.

[0002] Technical Field This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to block-level window size updates for arithmetic coding. [Background technology]

[0003] This background discussion provided herein is intended to generally present the context of the present disclosure. Work of the inventors named in this application, to the extent that their work is described in this background section, and aspects of this description that may not otherwise qualify as prior art at the time of the filing of this application, are not admitted, either explicitly or implicitly, as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated full or subsampled chrominance samples. The sequence of pictures can have a fixed or variable picture rate (alternatively referred to as frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames per second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video encoding and decoding may be to reduce redundancy in an uncompressed input video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process where the original video information is not fully preserved during encoding and is not fully recoverable during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application, even with some information loss. For video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, users of some consumer video streaming applications may tolerate higher distortion than users of cinema or television broadcast applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances: higher tolerable distortion generally allows for an encoding algorithm that results in higher loss and a higher compression ratio.

[0006] Video encoders and decoders may utilize techniques from several broad categories and stages, including motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be called an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of a block after intra prediction can be subjected to a transform to the frequency domain, and the transform coefficients so produced can be quantized before entropy coding. Intra prediction refers to a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, e.g. as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block to be intra-coded or decoded based on surrounding sample data and / or metadata preceding it in the decoding order, e.g. obtained during the encoding and / or decoding of spatial neighbors. Such techniques are referred to below as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed and not from other reference pictures.

[0009] There may be various forms of intra prediction. If more than one such technique is available in a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters for a block of video may be coded separately or may be included together in a mode codeword. Which codeword to use for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra prediction, as well as the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, refined in H.265, and further refined in newer coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, for intra prediction, a predictor block can be formed using neighborhood sample values ​​that are already available. For example, the available values ​​of a particular set of neighborhood samples along a certain direction and / or line may be copied into the predictor block. The reference to the direction used can be coded in the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, at the bottom right, a subset of nine predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes out of the 35 intra modes specified in H.265) is depicted. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent which direction neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from neighboring sample(s) to the top right and at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from neighboring sample(s) to the bottom left of sample (101), at an angle of 22.5 degrees from the horizontal.

[0012] Continuing to refer to FIG. 1A, at the top left, a square block (104) of 4×4 samples is depicted (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with an “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is at the bottom right. Additionally, an exemplary reference sample is shown that follows a similar numbering scheme. The reference sample is labeled R and its Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, predicted samples from neighboring neighborhoods adjacent to the block being reconstructed are used.

[0013] Intra-picture prediction of block 104 may begin by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block 104. That is, the sample is predicted from the top right predicted sample(s) at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then sample S44 is predicted from reference sample R08.

[0014] In some cases, particularly when the orientation is not divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding techniques continue to develop, the number of possible directions has increased. In H.264 (2003), for example, nine different directions are available for intra prediction. This increases to 33 in H.265 (2013), and JEM / VVC / BMS at the time of this disclosure can support up to 65 directions. Experimental studies have been performed to help identify the most preferred intra prediction direction, and certain techniques in entropy coding may be used to encode the most preferred direction with a small number of bits while accepting a bit penalty for the directions. Furthermore, sometimes the direction itself can be predicted from nearby directions used in intra prediction of neighboring blocks that have already been decoded.

[0016] FIG. 1B shows a schematic diagram (180) depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0017] The way in which the bits representing the intra-prediction directions are mapped to prediction directions in the coded video bitstream can vary from one video coding technique to another, ranging from a simple direct mapping of prediction directions to intra-prediction modes to complex adaptation schemes involving codewords, most-probable modes, and similar techniques. In any case, however, there may be certain intra-prediction directions that are statistically less likely to occur in the video content than certain other directions. Since a goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented by more bits than the more likely directions.

[0018] Inter- or inter-prediction may be based on motion compensation, in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) may be used for prediction of a newly reconstructed picture or part thereof (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. MV may have two dimensions, X and Y, or three dimensions, with the third dimension being an indication of the reference picture to be used (like a temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a region of sample data can be predicted from other MVs, e.g., from other MVs associated with other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in decoding order. Doing so can significantly reduce the overall amount of data required to encode the MV by relying on removing redundancy in correlated MVs, thereby increasing compression efficiency. MV prediction can work in a directed manner because, for example, when encoding an input video signal derived from a camera (known as natural video), there is a statistical probability that regions larger than the region to which a single MV is applicable move in a similar direction in the video sequence and can therefore, in some cases, be predicted using similar motion vectors derived from MVs of nearby regions. The result is that the actual MV for a given region is similar or identical to the MV predicted from the surrounding MVs. Such an MV can then be represented after entropy coding with fewer bits than would be used if the MV were directly encoded rather than predicted from the nearby MV(s). In some cases, the MV prediction may be an instance of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors in computing the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, a technique hereafter called "spatial merge" is described below.

[0021] Specifically, referring to Figure 2, a current block (201) contains samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly encoding its MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using the MV associated with any of the five surrounding samples, denoted A0, A1, and B0, B1, B2 (202 through 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks use.

[0022] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by building on the VP9 code base and incorporating additional techniques. The AV1 bitstream specification includes reference video codecs such as H.265 or the High Efficiency Video Coding (HEVC) standard or Versatile Video Coding (VVC). Summary of the Invention [Means for solving the problem]

[0023] The embodiments of the present disclosure provide a method and apparatus for video coding with block-level window size updates for arithmetic coding. The probability update rate may be a function of a count of occurrences for an associated syntax / context when parsing a bitstream. It may also be a function of the number of different symbol values ​​for the associated syntax / context. The value of the count is restarted at a predefined value at the block level. The probability or cumulative distribution function (CDF) update rate for a particular syntax / context may be reinitialized with a predefined value at the block level.

[0024] In one embodiment, a method for window size updates when decoding a video stream includes determining a count for a number of syntaxes when updating a window size; calculating a probability update rate as a function of the count for the associated syntax; and restarting the value of the count with a predefined value at a block level. The probability update rate is a function of the number of distinct symbol values ​​for the associated syntax. The method includes copying the probability values ​​of the syntaxes from another frame to a current frame; and restarting each value of the count for each frame with the predefined value. The restart is performed at a coding unit (CU) level, a prediction unit (PU) level, a transform unit (TU) level, a coding tree unit (CTU) level, or a specified block size level. The block size for restart is specified in a high level syntax including a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter setting (SPS), an adaptation parameter set (APS), a picture header, a frame header, a slice header, a tile header, or a coding tree unit (CTU) header. The predefined values ​​of count after restart include values ​​of 2n, where n is an integer. Restart is syntax dependent and the probability update rate is controlled differently for different syntaxes. The probability update rate applies to a specific syntax group and thus the probability update rate is restarted at different times depending on the syntax group and the probability update rate is restarted at different values ​​depending on the syntax group. A flag is signaled at the block level to indicate whether the selected syntax group restarts the counter to update the probability update rate.

[0025] In another embodiment, an apparatus for decoding a video bitstream includes a memory storing instructions and a processor in communication with the memory, where when the processor executes the instructions, the processor is configured to cause the apparatus to: maintain a count of occurrences of a context; determine a cumulative distribution function (CDF) update rate for the context based on the count; and reinitialize the CDF with a predefined value for the count at a block level. The reinitialization is performed at a coding unit (CU) level, a prediction unit (PU) level, a transform unit (TU) level, a coding tree unit (CTU) level, or a specified block size level. The block size for reinitialization includes a high level syntax, where the high level syntax includes a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter setting (SPS), an adaptation parameter set (APS), a picture header, a frame header, a slice header, a tile header, or a coding tree unit (CTU) header. The predefined value includes a value corresponding to a uniform distribution. The predefined value is stored at a given instance during encoding or decoding. Reinitialization of the CDF is context dependent, and the CDF is controlled differently for different contexts. Reinitialization is performed for a particular context group. Reinitialization occurs at different times depending on the context group, and with different values ​​depending on the context group. A flag is signaled at the block level to indicate whether a selected context group reinitializes a counter to update the CDF. Contexts are grouped into different sets, each set can reinitialize the CDF together and is controlled by one flag. The processor is configured to cause the device to: copy a CDF update rate from another frame to the current frame; and restart each value of the count for each frame with the predefined value.

[0026] In some other embodiments, a device for processing video information is disclosed. The device may include circuitry configured to perform any one of the implementations described above.

[0027] Embodiments of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform video decoding and / or encoding. [Brief description of the drawings]

[0028] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0029] [Figure 1A] 1 shows a schematic illustration of an exemplary subset of intra-prediction directional modes.

[0030] [Figure 1B] 1 illustrates an illustration of an exemplary intra-prediction direction.

[0031] [Diagram 2] 1 shows a schematic illustration of a current block and its surrounding spatial merging candidates for motion vector prediction in an example.

[0032] [Diagram 3] 1 shows a schematic illustration of a simplified block diagram of a communication system according to an example embodiment;

[0033] [Figure 4] 1 shows a schematic illustration of a simplified block diagram of a communication system (400) according to an example embodiment.

[0034] [Diagram 5] 1 shows a schematic illustration of a simplified block diagram of a video decoder in accordance with an example embodiment;

[0035] [Figure 6] 1 shows a schematic illustration of a simplified block diagram of a video encoder according to an example embodiment;

[0036] [Figure 7] 1 shows a block diagram of a video encoder according to another exemplary embodiment.

[0037] [Figure 8] 1 shows a block diagram of a video decoder according to another exemplary embodiment.

[0038] [Figure 9] 1 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure.

[0039] [Figure 10] 1 illustrates another coding block partitioning scheme according to an exemplary embodiment of the present disclosure.

[0040] [Figure 11] 1 illustrates another coding block partitioning scheme according to an exemplary embodiment of the present disclosure.

[0041] [Figure 12] 1 illustrates an exemplary partitioning of a base block into coding blocks according to an exemplary partitioning scheme.

[0042] [Figure 13] 1 illustrates an exemplary ternary partitioning scheme.

[0043] [Figure 14] 1 illustrates an exemplary Quad-Tree Binary Tree (QTBT) coding block partitioning scheme.

[0044] [Figure 15]4 illustrates a scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the present disclosure.

[0045] [Figure 16] 4 illustrates another scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the present disclosure.

[0046] [Figure 17] 4 illustrates another scheme for partitioning a coding block into multiple transform blocks, according to an exemplary embodiment of the present disclosure.

[0047] [Figure 18] 1 shows a flowchart for an exemplary code for a single binary decision.

[0048] [Figure 19A] 1 illustrates a flowchart of a method according to an example embodiment of the present disclosure.

[0049] [Figure 19B] 4 shows a flowchart of a method according to another exemplary embodiment of the present disclosure.

[0050] [Figure 20] 1 shows a schematic illustration of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0051] Throughout the specification and claims, terms may have a nuanced meaning suggested or implied in the context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include combinations of the example embodiments / implementations in whole or in part.

[0052] Generally, terms may be understood, at least in part, from their use in context. For example, terms such as "and," "or," or "and / or," as used herein, may include a variety of meanings that may depend, at least in part, on the context in which such terms are used. Typically, "or," when used to relate a list such as A, B, or C, is intended to mean A, B, and C, as used herein in an inclusive sense, as well as A, B, or C, as used herein in an exclusive sense. Furthermore, the terms "one or more" or "at least one," as used herein, may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, at least in part, depending on the context. Similarly, terms such as "a," "an," or "the" may be understood to convey a singular use or to convey a plural use, at least in part, depending on the context. Moreover, the terms "based on" or "determined by" are not necessarily intended to convey an exclusive set of factors, but instead may permit the existence of additional factors not necessarily explicitly recited, again depending at least in part on the context.

[0053] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., video data of a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover a video picture, and display the video picture according to the recovered video data. The one-way data transmission may be implemented in a media service application, etc.

[0054] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of encoded video data, such as may be implemented during a video conferencing application. For the bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., video data of a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0055] In the example of FIG. 3, terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, although applicability of the principles underlying the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or others. Network (350) represents any number or type of network that conveys encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication network (350) may exchange data in circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present discussion, the architecture and topology of network (350) may not be important to the operation of the present disclosure unless explicitly described.

[0056] 4 illustrates an arrangement of a video encoder and a video decoder in a video streaming environment as an example application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0057] The video streaming system may include a video source (401), for example a digital camera, and may include a video capture subsystem (413) that generates a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples encoded by the digital camera of the video source 401. The stream of video pictures (402), depicted as a thick line to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or achieve aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize its lower data volume compared to the stream of uncompressed video pictures (402), can be stored on the streaming server (405) for future use or directly stored on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example within the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) and generates an outgoing stream of uncompressed video pictures (411) that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) may be encoded according to some video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video encoding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC or other video encoding standards.

[0058] It is noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0059] 5 shows a block diagram of a video decoder (510) according to any of the embodiments of the present disclosure described below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (310) in the example of FIG. 4.

[0060] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of the other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the encoded video data or a streaming source that transmits the encoded video data. The receiver (531) may receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded through respective processing circuits (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereafter referred to as the "parser"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be external to the video decoder (510) and separate from the video decoder (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g. to combat network jitter, and there may be another additional buffer memory (515) internal to the video decoder (510), e.g. to handle playback timing. If the receiver (531) is receiving data from a storage / forwarding device of sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may not be needed or may be small.For use with best-effort packet networks such as the Internet, a buffer memory (515) of sufficient size may be required and may be relatively large in size. Such a buffer memory may be implemented with an adaptive size and may be implemented, at least in part, external to the video decoder (510) in an operating system or similar element (not shown).

[0061] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a display (512) (e.g., a display screen). The rendering device may be an integral part of the electronic device (530) as shown in FIG. 5, or may be coupled to the electronic device (530) rather than being an integral part. The control information for the rendering device(s) may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may follow a video coding technique or standard and may follow a variety of principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on the at least one parameter corresponding to the subgroup. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0062] The parser (520) performs an entropy decoding / parsing operation on the video sequence received from the buffer memory (515), thereby generating symbols (521).

[0063] The reconstruction of the symbols (521) may involve several different processing or functional units, depending on the type of coded video picture or portions thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. The units involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and the several processing or functional units described below is not depicted for simplicity.

[0064] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually divided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the remainder of this disclosure.

[0065] The first unit may include a scaler / inverse transform unit (551), which receives quantized transform coefficients and control information as symbol(s) (521) from the parser (520). The control information includes information indicating what type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, etc. The scaler / inverse transform unit (551) may output a block containing sample values ​​that can be input to an aggregator (555).

[0066] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed using already reconstructed surrounding block information currently stored in the picture buffer (558). The current picture buffer (558) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. Depending on the implementation, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0067] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such a case, the motion compensation prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols (521) for the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (the output of unit 551 may also be referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensation unit (553) fetches the prediction samples may be controlled by a motion vector available to the motion compensation unit (553) in the form of symbols (521). The symbols may have, for example, X, Y components (shift) and a reference picture component (time). Motion compensation may involve interpolation of sample values ​​taken from a reference picture memory (557) when sub-sample accurate motion vectors are used, may be associated with a motion vector prediction mechanism, etc.

[0068] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, as well as to previously reconstructed loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in more detail below.

[0069] The output of the loop filter unit (556) can be a sample stream, which can be output to a rendering device (512) or can be stored in a reference picture memory (557) for use in future inter-picture prediction.

[0070] Once a coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557) and a fresh current picture buffer can be reallocated before starting the reconstruction of the subsequent coded picture.

[0071] The video decoder (510) may perform decoding operations according to a given video compression technique adopted in a standard such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax prescribed by the video compression technique or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select certain tools from all tools available in the video compression technique or standard as the only tools available for use under that profile. To conform to a standard, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in units of megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management, possibly signaled in the coded video sequence.

[0072] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) improvement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0073] 6 illustrates a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.

[0074] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture video images to be encoded by the video encoder (603). In another example, the video source (601) can be implemented as part of the electronic device (620).

[0075] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, XYZ ...) and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device that can store pre-prepared videos. In a video conferencing system, the video source (601) may be a camera that captures image information locally as a video sequence. The video data may be provided as a number of individual pictures or images that impart motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0076] According to some example embodiments, the video encoder (603) can encode and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under any other time constraint required by an application. Enforcing an appropriate encoding rate is one function of the controller (650). In some embodiments, the controller (650) may be operatively coupled to and control other functional units, such as those described below. Such couplings are not depicted for simplicity. Parameters set by the controller (650) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions for the video encoder (603) optimized for a certain system design.

[0077] In some example embodiments, the video encoder (603) may be configured to operate in an encoding loop. As a simplistic explanation, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for generating symbols such as a symbol stream based on an input picture to be encoded and a reference picture(s)) and a (local) decoder (633) embedded in the video encoder (603). The embedded decoder 633 processes the encoded video stream by the source encoder 630 without entropy encoding, but the decoder (633) reconstructs the symbols to generate sample data in a manner similar to how a (remote) decoder would generate them (in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream in the entropy encoding may be lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream yields bit-accurate results regardless of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using the prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0078] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, e.g., the video decoder (410), already described in detail above in connection with Figure 5. However, referring also briefly to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (420) may be lossless, the entropy decoding portion of the video decoder (410), including the buffer memory (415) and the parser (420), may not be fully implemented in the local decoder (633) at the encoder.

[0079] An observation that can be made at this point is that any decoder technique, except for parsing / entropy decoding, which may only be present in the decoder, may need to be present in substantially the same functional form in the corresponding encoder. For this reason, although the disclosed subject matter may at times focus on decoder operations, this also applies to the decoding portion of an encoder. In this manner, the description of the encoder technique may be shortened, since it is the inverse of the decoder technique, which is described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.

[0080] In operation, in some example implementations, the source encoder (630) may perform motion-compensated predictive encoding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence, designated as "reference pictures." In this manner, the encoding engine (632) encodes differences (or residuals) in color channels between pixel blocks of the input picture and pixel blocks of a reference picture(s) that may be selected as a predictive reference for the input picture. The terms "residual" and its attributive form "residual" may be used interchangeably.

[0081] The local video decoder (633) can decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source encoder (630). The operation of the coding engine (632) can advantageously be a lossy process. When the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can be a copy of the source video sequence, typically with some errors. The local video decoder (633) can replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content (in the absence of transmission errors) as the reconstructed reference pictures that would be obtained by the far-end (remote) video decoder.

[0082] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (634).

[0083] The controller (650) may manage the encoding operations of the source encoder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0084] The output of all the above functional units can be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0085] The transmitter (640) can buffer the coded video sequence produced by the entropy encoder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the coded video data from the video encoder (630) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0086] A controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a coding picture type to each coded picture. The coding picture type may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0087] An intra picture (I picture) may be one that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art will recognize these variations of I pictures, as well as their respective uses and characteristics.

[0088] A predictive picture (P picture) may be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0089] A bidirectionally predictive picture (B picture) may be one that can be encoded and decoded using intra- or inter-prediction that uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predictive picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0090] A source picture is usually spatially divided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. A source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same manner as detailed further below.

[0091] The video encoder (603) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0092] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0093] Video may be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits correlations (temporal or otherwise) between pictures. For example, a particular picture to be encoded / decoded, called the current picture, may be divided into blocks. If a block in the current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0094] In some exemplary embodiments, bi-prediction techniques can be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be past or future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be jointly predicted by a combination of the first and second reference blocks.

[0095] Furthermore, to improve coding efficiency, merge mode techniques may be used in inter-picture prediction.

[0096] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU may be partitioned into one CU of 64×64 pixels or four CUs of 32×32 pixels. Each of the one or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some example embodiments, each CU is analyzed during encoding to determine a prediction type for that CU, such as an inter prediction type or an intra prediction type, among various prediction types. A CU may be divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in encoding / decoding are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of values ​​(e.g., luma values) for samples, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0097] 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. This exemplary video encoder (703) may be used in place of the video encoder (403) in the example of FIG. 4.

[0098] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block, such as 8×8 samples. The video encoder (703) then determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, e.g., using rate-distortion optimization (RDO). If it is determined that the processing block is coded in intra-mode, the video encoder (703) may use intra-prediction techniques to encode the processing block into a coded picture; if it is determined that the processing block is coded in inter-mode or bi-prediction mode, the video encoder (703) may use inter-prediction techniques or bi-prediction techniques, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, a merge mode may be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors, but without the benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the current block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module for determining a prediction mode for a processing block.

[0099] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy coder (725), coupled together as shown in the exemplary arrangement in FIG.

[0100] The inter encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter prediction information (e.g., a description of redundant information from an inter encoding technique, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a predicted block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on encoded video information using a decoding unit 633 (shown as residual decoder 728 in FIG. 7 ) embedded in the example encoder 620 of FIG. 6 , which is described in more detail below.

[0101] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to blocks already encoded in the same picture, generate transformed quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder (722) may calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and reference blocks in the same picture.

[0102] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode for a block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is an intra mode, the general controller (721) controls the switch (726) to select a result of the intra mode for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; if the prediction mode for the block is an inter mode, the general controller (721) controls the switch (726) to select a result of inter prediction for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0103] The residual calculator (723) may be configured to calculate a difference (residual data) between a received block and a prediction result for that block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data may be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks may be suitably processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures.

[0104] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When encoding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual information may not be present.

[0105] 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0106] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in the example arrangement of FIG. 8.

[0107] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent the syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information, etc.) that may identify the mode in which the block is coded (e.g., intra- or inter-prediction mode, merged submode, or another submode), certain samples or metadata used for prediction by the intra- or inter-decoder (872) or inter-decoder (880), residual information, e.g., in the form of quantized transform coefficients, etc. In one example, if the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880); if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0108] An inter decoder (880) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0109] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0110] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (871) (data path not depicted since this is only low data volume control information).

[0111] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual output by the residual decoder (873) and the prediction result (output by the intra- or inter-prediction module, as the case may be) to form a reconstructed block, which forms part of the reconstructed picture as part of the reconstructed video. Note that other suitable operations, such as deblocking operations, may also be performed to improve visual quality.

[0112] It should be noted that the video encoders (403), (603), (703) and the video decoders (410), (510), (810) may be implemented using any suitable technique. In some exemplary embodiments, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), (603) and the video decoders (410), (510), (810) may be implemented using one or more processors executing software instructions.

[0113] Turning to block partitioning for encoding and decoding, a general partitioning may start from a base block and follow a predefined set of rules, specific patterns, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After dividing or partitioning the base block according to any of the exemplary partitioning procedures described below or other procedures, or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each partition may be referred to as a coding block (CB). For various exemplary partitioning implementations described further below, each resulting CB may be of any of the allowable sizes and partitioning levels. Such partitions are referred to as coding blocks because they may be units at which some basic encoding / decoding decisions may be made and encoding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The top or deepest level of the final partitions represents the depth of the coding block partitioning structure of the tree. A coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be referred to as a coding block tree (CBT).

[0114] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure for various color channels within a CTU may or may not be the same.

[0115] In some implementations, the partition tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same or different coding partition tree structures and the actual coding partition tree structure used may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0116] In some example implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an example 4-way partition tree may start from a first predefined level (e.g., 64×64 block level or other size as the base block size), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., 4×4 level). For example, the base block may follow four predefined partitioning options or patterns as shown by 902, 904, 906, and 908, and the same partitioning options as shown in FIG. 9 may be repeated at a lower scale, down to the lowest level (e.g., 4×4 level). In some implementations, additional constraints may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but rectangular partitions may not be allowed to be recursive. On the other hand, square partitions are allowed to be recursive. Partitioning according to FIG. 9, with recursion if necessary, produces a final set of coding blocks. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth for the root node or root block, e.g., a 64x64 block, may be set to 0, and once the root block is further partitioned according to FIG. 9, the coding tree depth is increased by 1. The maximum or deepest level from the 64x64 base block to the 4x4 minimum partition is 4 for the above scheme (starting from level 0). Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be partitioned independently according to the scheme of FIG. 9 (e.g., a partitioning pattern or option among predefined patterns may be determined independently for each of the color channels at each hierarchical level).Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partition pattern or option of the predefined patterns may be selected for two or more color channels at each hierarchical level).

[0117] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (e.g., from a base block at a 128×128 level or a 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. A partition type with three subpartitions, denoted 1002, 1004, 1006, and 1008, shown in the second row of FIG. 10, may be referred to as a “T” partition. The “T” partitions 1002, 1004, 1006, and 1008 may be referred to as left T, upper T, right T, and lower T. In some exemplary implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth for the root node or root block, e.g., a 128x128 block, may be set to 0, and once the root block is further partitioned according to FIG. 10, the coding tree depth is increased by 1. In some implementations, only the all-square partitions in 1010 may be allowed to be recursively partitioned to the next level of the partitioning tree according to the pattern of FIG. 10. In other words, for the square partitions in the T-shaped patterns 1002, 1004, 1006, and 1008, recursive partitioning may not be allowed. The partitioning procedure according to FIG. 10, with recursion if necessary, generates a final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, further flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0118] In some other example implementations for coding block partitioning, a quadtree structure may be used to partition a base block or an intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square partitions. Whether a base block or intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of that base block or intermediate block / partition. The quadtree partitioning at a picture boundary may be further adapted. For example, an implicit quadtree partitioning may be performed at the picture boundary, whereby a block may continue to be quadtree partitioned until its size fits into the picture boundary.

[0119] In some other example implementations, a hierarchical binary partitioning from a base block may be used. For such a scheme, a base block or a mid-level block may be divided into two partitions. The binary partitioning may be horizontal or vertical. For example, a horizontal binary partitioning may divide a base block or a mid-level block into equal left and right partitions. Similarly, a vertical binary partitioning may divide a base block or a mid-level block into equal top and bottom partitions. Such binary partitioning may be hierarchical and recursive. At each of the base block or mid-level blocks, a determination may be made as to whether the binary partitioning scheme should continue and, if the scheme continues further, whether a horizontal or vertical binary partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop once a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be constrained. For example, the aspect ratio of a partition is no smaller than 1:4 (or no larger than 4:1). Thus, a vertical strip partition with a vertical-to-horizontal aspect ratio of 4:1 can only be further bisected vertically into upper and lower partitions with vertical-to-horizontal aspect ratios of 2:1 each.

[0120] In still other examples, a ternary partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown at 1302 in FIG. 13, or horizontally, as shown at 1304 in FIG. 13. The exemplary partitioning ratio in FIG. 13 is shown as 1:2:1, both vertically and horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary partitioning scheme may be used to complement a quadtree or binary partitioning structure, such that a ternary tree partitioning can capture objects located at the block center in one contiguous partition, whereas quadtrees and binary trees always split along the block center, thus splitting the objects into separate partitions. In some implementations, the width and height of the exemplary ternary tree partitions are always powers of two to avoid additional transformations.

[0121] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-mentioned quadtree and binary partitioning schemes may be combined to partition the base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be quadtree-partitioned or binary-partitioned according to a set of predefined conditions, if specified. A specific example is shown in FIG. 14.

[0122] FIG. 14 illustrates an example partition and tree for a quad-tree plus binary tree (QTBT) structure. The QTBT structure may not have a similar concept of multiple partition types and may eliminate the separation of the concepts of CU, PU, ​​and TU. The QTBT structure may support increased flexibility for CU partition shapes. In some embodiments of the QTBT block structure, CUs may have a square or rectangular shape.

[0123]

[0124] In the example of FIG. 14, the base block is first quadtree partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each resulting partition is then either quadtree partitioned at the next level into four further partitions (such as 1408), bipartitely split (horizontally or vertically) into two further partitions (such as 1402 or 1406; either symmetrically, for example), or not split (such as 1404). For square partitions, bipartitioning or quadtree partitioning may be allowed recursively, as shown by the overall exemplary partition pattern at 1410 and the corresponding tree structure / representation at 1420, where solid lines represent quaternary partitions and dashed lines represent binary partitions. For each binary partition node (non-leaf binary partitions), a flag may be used to indicate whether the binary partition is horizontal or vertical. For example, as shown at 1420, consistent with the partitioning structure of 1410, a flag "0" may represent a horizontal bisection and a flag "1" may represent a vertical bisection. For quadtree partitions, there is no need to indicate the partition type, since quadtree partitioning always splits a block or partition both horizontally and vertically, generating four sub-blocks / partitions of equal size. In some implementations, a flag "1" may represent a horizontal bisection and a flag "0" may represent a vertical bisection.

[0125] In the example shown in FIG. 14, the coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. There are two types of binary tree partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called coding units (CUs), and their segmentation can be used for prediction and transform processing without further partitioning. In the QTBT coding block structure, CUs, PUs, and TUs may have the same block size. In JEM, a CU may contain coding blocks (CBs) of different color components (e.g., for P and B slices in 4:2:0 chroma format, one CU contains one luma CB and two luma CBs). In other embodiments, it may contain a single component CB (e.g., for I slices, one CU contains one luma CB or two chroma CBs).

[0126] In some example implementations of QTBT, the quadtree and binary partitioning rule sets may be represented by the following predefined parameters and their associated corresponding functions: CTU size: Root node size of the quadtree (size of the base block) MinQTSize: The minimum allowed quadtree leaf node size. · MaxBTSize: The maximum allowed binary tree root node size. · MaxBTDepth: Maximum allowed binary tree depth MinBTSize: The minimum allowed binary tree leaf node size. In some exemplary implementations of the QTBT partitioning structure, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (if exemplary chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (for both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. Quad-tree partitioning may be applied to the CTU first to generate quad-tree leaf nodes. The quad-tree leaf nodes may have sizes ranging from the minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a node is 128x128, it will not be split by the binary tree first because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a binary tree. In the example of Figure 14, the base block is 128x128. The base block can only be quadtree partitioned according to a predefined set of rules. The base block has a partition depth of 0. Each of the resulting four partitions is 64x64, does not exceed MaxBTSize, and can be further quadtree or binary tree partitioned at level 1. The process continues. If the depth of the binary tree reaches MaxBTDepth (i.e. 4), no further partitions may be considered. If a binary tree node has a width equal to MinBTSize (i.e. 4), no further horizontal partitions may be considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical partitions may be considered.

[0127] In some example implementations, the above QTBT scheme may be configured to support the flexibility of luma and chroma having the same or separate QTBT structures. For example, for P and B slices, the luma CTB and chroma CTB in one CTU may share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by another QTB structure. This means that a CU may be used to refer to different color channels in an I slice. For example, an I slice may consist of a coding block of a luma component, or a coding block of two chroma components, and a CU in a P or B slice may consist of coding blocks of all three color components.

[0128] Figure 14 shows an example of block partitioning by using QTBT (left side of Figure 14) and the corresponding tree representation (right side of Figure 14). Solid lines indicate quadtree partitioning and dotted lines indicate binary tree partitioning. At each partition (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For quadtree partitioning, there is no need to indicate the partition type since quadtree partitioning partitions the block both horizontally and vertically to generate four sub-blocks of equal size.

[0129] In some other implementations, the QTBT scheme may be supplemented with the ternary partitioning scheme described above. Such implementations may be referred to as multi-type-tree (MTT) structures. For example, in addition to the bipartite division of nodes, one of the ternary partitioning patterns of FIG. 13 may be selected. In some implementations, only square nodes may undergo ternary partitioning. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0130] Two-level or multilevel tree designs such as the QTBT implementation and the QTBT implementation supplemented by a trichotomous partition can be motivated primarily by reduced complexity. In theory, the complexity of traversing a tree is T D where T represents the number of partition types and D is the depth of the tree. Tradeoffs can be made by using multiple types (T) while keeping the depth (D) small.

[0131] Furthermore, the QTBT scheme supports the flexibility of luma and chroma having separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU share the same QTBT structure. However, for I slices, the luma CTB is partitioned into CUs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by another QTBT structure. In this example, a CU in an I slice contains a coded block of a luma component or a coded block of two chroma components, and a CU in a P or B slice contains a coded block of all three color components. In HEVC, to reduce memory accesses for motion compensation, inter prediction for small blocks may be constrained, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks. In the QTBT implemented in JEM-7.0, these constraints can be removed.

[0132] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for the purpose of intraframe or interframe prediction during the encoding and decoding process. In other words, the CB may be further partitioned into different subpartitions where individual prediction decisions / configurations may be made. In parallel, the CB may be further partitioned into multiple transform blocks (TBs) to delineate the level at which the transformation or inverse transformation of the video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The partitioning scheme of the PBs and TBs may be independent in some exemplary implementations. The partitioning schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be partitioned after the PB partitions, and in particular, each PB after being determined after the partitioning of the coding blocks may be further partitioned into one or more TBs. For example, in some implementations, the PB may be divided into 1, 2, 4, or some other number of TBs.

[0133] In some implementations, for the purpose of partitioning base blocks into coding blocks and further into prediction and / or transform blocks, the luma and chroma channels may be treated differently. For example, in some implementations, partitioning of coding blocks into prediction and / or transform blocks may be allowed for the luma channel, while such partitioning of coding blocks into prediction and / or transform blocks may not be allowed for the chroma channel(s). In such implementations, transformation and / or prediction of luma blocks may be performed only at the coding block level. In another example, the minimum transform block size for the luma and chroma channel(s) may be different. For example, coding blocks for the luma channel may be allowed to be partitioned into smaller transform and / or prediction blocks than the chroma channels. In yet another example, the maximum depth of partitioning of a coding block into transform and / or prediction blocks may be different between luma and chroma channels, e.g., a coding block for a luma channel may be allowed to be partitioned into deeper transform and / or prediction blocks than the chroma channel(s). In a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes that can be represented by recursive partitions going down up to two levels, and transform block shapes such as square, 2:1 / 1:2, 4:1 / 1:4, etc., and transform block sizes from 4×4 to 64×64 may be allowed. However, for a chroma block, only the largest possible transform block specified for the luma block may be allowed.

[0134] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0135] Partitioning of coding blocks (or prediction blocks) into transform blocks may be realized in various exemplary manners. Such manners include, but are not limited to, quadtree partitioning and predefined pattern partitioning, recursive or non-recursive, and may take additional consideration of transform blocks at boundaries of coding blocks or prediction blocks. In general, the resulting transform blocks may be at different partitioning levels, may not be the same size, and may not be square shaped (e.g., may be rectangular with an acceptable size and aspect ratio). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0136] However, in some other implementations, the CB obtained via any of the above partitioning schemes may be used as a basic or smallest coding block for prediction and / or transformation. In other words, no further partitioning is performed for inter / intra prediction purposes and / or transformation purposes. For example, the CB obtained from the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU and TU, and supports more flexibility for the partition shapes of CU / CB, as described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without further partitioning. This means that in such an exemplary QTBT coding block structure, CU, PU, ​​TU have the same block size.

[0137] The various CB partitioning schemes described above and further partitioning of the CB into PBs and / or TBs (including no PB / TB partitioning) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0138] Specific exemplary implementations of coding block and transform block partitioning are described below. In such exemplary implementations, a base block may be partitioned into coding blocks using the recursive quadtree partitioning described above (such as those in FIG. 9 and FIG. 10) or a predefined partitioning pattern. At each level, whether further quadtree partitioning of a particular partition should continue may be determined by local video data characteristics. The resulting CBs may be of different sizes at different quadtree partitioning levels. The decision of whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction may be made at the CB level (or at the CU level for all three color channels). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partitioning type. Inside one PB, the same prediction process may be applied, and related information may be transmitted to the decoder for each PB. After obtaining the residual block by applying the prediction process based on the PB partitioning type, the CB may be partitioned into TBs according to another quadtree structure similar to the coding tree for the CB. In this particular implementation, the CB or TB may be square, but need not be limited thereto. Furthermore, in this particular example, the PB may be square or rectangular in shape for inter prediction, and may only be square for intra prediction. The coding block may be divided, for example, into four square TBs. Each TB may be further divided recursively (using quadtree division) into smaller TBs, called residual quadtrees (RQTs).

[0139] Another exemplary implementation for partitioning the base block into CB, PB, and / or TB is described further below. For example, rather than using multiple partition unit types as shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type trees using bisection and trisection segmentation structures (e.g., QTBT, or QTBT with trisection, as described above) may be used. Separation of CB, PB, and TB (i.e., partitioning CB into PB and / or TB, and partitioning PB into TB) may be abandoned unless required for CBs with sizes too large for the maximum transform length (such CBs may need to be further divided). This exemplary partitioning scheme may be designed to support further flexibility for CB partition shapes, such that prediction and transformation can both be performed at the CB level without further partitioning. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, the coding tree block (CTB) is first partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by a nested multitype tree structure. An example of a nested multitype tree structure using bisection or trisection is shown in FIG. 11. Specifically, the example multitype tree structure of FIG. 11 includes four partition types called vertical bisection (SPLIT_BT_VER) (1102), horizontal bisection (SPLIT_BT_HOR) (1104), vertical trisection (SPLIT_TT_VER) (1106), and horizontal trisection (SPLIT_TT_HOR) (1108). And CB corresponds to the leaf of the multitype tree. In this example implementation, this segmentation is used for both prediction and transform processing without further partitioning, as long as CB is not too large for the maximum transform length. This means that in most cases, in a quadtree with a nested multitype tree coding block structure, CB, PB, and TB have the same block size.An exception occurs when the maximum supported transform length is smaller than the width or height of a color component of the CB. In some implementations, in addition to the bisection or trisection division, the nested pattern of Figure 11 may further include a quadtree division.

[0140] One example of a quadtree with nested multi-type tree coding block structure of block partitions for one base block (including quadtree, bisection, and trisection split options) is shown in FIG. 12. More specifically, FIG. 12 shows a base block 1200 being quadtree split into four square partitions 1202, 1204, 1206, and 1208. A decision is made for each quadtree split partition to further use the multi-type tree structure and quadtree of FIG. 11 for further splitting. In the example of FIG. 12, partition 1204 is not further split. Partitions 1202 and 1208 each employ another round of quadtree splitting. For partition 1202, the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt the third level quadtree partitioning, horizontal bisection partition 1104 of FIG. 11, unsplit, and horizontal thirds partition 1108 of FIG. 11, respectively. Partition 1208 adopts another quadtree partitioning, and the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt the third level vertical thirds partition 1106 of FIG. 11, unsplit, unsplit, and horizontal bisection partition 1104 of FIG. 11, respectively. Two of the subpartitions of the third level top-left partition of 1208 are further partitioned according to the horizontal bisection partition 1104 and horizontal thirds partition 1108 of FIG. 11, respectively. Partition 1206 adopts a second level division pattern into two partitions according to vertical bisection division 1102 of Figure 11, which are further divided at a third level according to horizontal third division 1108 and vertical bisection division 1102 of Figure 11. Furthermore, a fourth level division is applied to one of them according to horizontal bisection division 1104 of Figure 11.

[0141] For the particular example above, the maximum luma transform size may be 64×64 and the maximum supported chroma transform size may be different from the luma, e.g., 32×32. Although the example CB above in Figure 12 is not generally further divided into smaller PBs and / or TBs, if the width or height of a luma or chroma coding block is larger than the maximum transform width or height, the luma or chroma coding block may be automatically divided in the horizontal and / or vertical directions to satisfy the transform size constraints in the horizontal and / or vertical directions.

[0142] In the particular example of partitioning base blocks into CBs above, as mentioned above, the coding tree scheme can support that luma and chroma can have separate block tree structures. For example, for P and B slices, the luma CTB and chroma CTB in one CTU may share the same coding tree structure. For I slices, for example, luma and chroma can have separate coding block tree structures. When the separate block tree structure is applied, the luma CTB is partitioned into luma CBs by one coding tree structure, and the chroma CTB is partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of the luma component, or a coding block of two chroma components, and a CU in a P or B slice is always composed of coding blocks of all three color components, unless the video is monochrome.

[0143] If a coding block is further divided into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning aspects. Exemplary implementations for partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as described above, the transform partitioning may support multiple shapes of blocks, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with the transform block sizes ranging from, e.g., 4×4 to 64×64. In some implementations, if the coding block is 64×64 or smaller, the transform block partitioning may be applied only to the luma component, and for the chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the width or height of the coding block is greater than 64, the luma and chroma coding blocks may be implicitly divided into multiple min(W,64)×min(H,64) and min(W,32)×min(H,32) transform blocks, respectively.

[0144] In some example implementations of transform block partitioning, for both intra- and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with a partitioning depth up to a predefined number of levels (e.g., two levels). The depth and size of the transform block partitioning may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1. [Table 1]

[0145] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partition can generate four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, the transform size for the current depth 4×4 corresponds to the same transform size of 4×4 for the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partition can generate two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform partition can generate two 1:2 / 2:1 sub-transform blocks.

[0146] In some example implementations, additional constraints may be applied regarding transform block partitioning for the luma component of an intra-coded block. For example, for each level of transform partitioning, all sub-transform blocks may be constrained to have equal size. For example, for a 32×16 coding block, the level 1 transform partitioning creates two 16×16 sub-transform blocks, and the level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform units equal in size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The first level partitioning into four equal-sized transform blocks according to Table 1 is shown in 1504, with the coding order indicated by the arrows. The second level division of all first level equal sized blocks into 16 equal sized transform blocks according to Table 1 is shown at 1506 with the coding order indicated by the arrows.

[0147] In some example implementations, the above constraints on intra-coding may not apply for the luma components of inter-coded blocks. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently for another level. Thus, the resulting transform blocks may or may not be of the same size. An example partitioning of an inter-coded block into transform blocks is shown in FIG. 16 along with the coding order. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one (but not all) of the four transform blocks is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as indicated by 1604. An example coding order of these seven transform blocks is indicated by the arrows at 1604 in FIG. 16.

[0148] In some example implementations, for a chroma component(s), some additional constraints on the transform block may be applied, e.g., for a chroma component(s), the transform block size may be the same size as the coding block size, but not smaller than a predefined size, e.g., 8×8.

[0149] In some other example implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure, “min(a,b)” can return the smaller value between a and b.

[0150] FIG. 17 further illustrates another alternative exemplary scheme for partitioning a coding block or a prediction block into transform blocks. As illustrated in FIG. 17, instead of using recursive transform partitioning, a predefined set of partitioning types may be applied to a coding block according to its transform type. In the particular example illustrated in FIG. 17, one of six exemplary partitioning types may be applied to partition a coding block into a varying number of transform blocks. Such a scheme for generating transform block partitioning may be applied to either a coding block or a prediction block.

[0151] More specifically, the partitioning scheme of FIG. 17 provides up to six example partition types for any given transform type (transform type refers to the type of primary transform, e.g., ADST and others). In this scheme, every coding block or predictive block may be assigned a transform partition type based on, e.g., a rate-distortion cost. In one example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to a transform block partition size and pattern, as illustrated by the six transform partition types illustrated in FIG. 17. The correspondence between the various transform types and the various transform partition types may be predefined. An example is illustrated below. The uppercase English labels indicate transform partition types that may be assigned to a coding block or predictive block based on a rate-distortion cost. ·PARTITION_NONE: Allocates the transformation size equal to the block size. ·PARTITION_SPLIT: Allocates a transformation size of 1 / 2 the width of the block size and 1 / 2 the height of the block size. ·PARTITION_HORZ: Allocates a transformation size with the same width as the block size and 1 / 2 the height of the block size. ·PARTITION_VERT: Allocates a transformation size that is 1 / 2 the width of the block size and the same height as the block size. ·PARTITION_HORZ4: Allocates a transformation size with the same width as the block size and 1 / 4 of the block size in height. ·PARTITION_VERT4: Allocates a transformation size that is 1 / 4 the width of the block size and the same height as the block size.

[0152] In the above example, all of the transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is by way of example only and not limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks in a particular partition type (or pattern).

[0153] The PBs (or CBs, also referred to as PBs when not further partitioned into predictive blocks) obtained from any of the above partitioning schemes may then become individual blocks for encoding via either intra-prediction or inter-prediction. For inter-prediction for the current PB, a residual between the current block and the predictive block may be generated, encoded, and included in the encoded bitstream.

[0154] Inter prediction may be implemented, for example, in single reference mode or mixed reference mode. In some implementations, a skip flag may first be included in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and should not be skipped. If the current block is inter-coded, another flag may further be included in the bitstream to signal whether a single reference mode or a mixed reference mode is used for prediction of the current block. For a single reference mode, one reference block may be used to generate a prediction block for the current block. For a mixed reference mode, two or more reference blocks may be used to generate a prediction block, for example by a weighted average. A mixed reference mode may be referred to as a two or more reference mode, a two-reference mode, or a multiple reference mode. The one or more reference blocks may be identified using one or more reference frame indices and further using one or more corresponding motion vectors indicating a shift(s) in position, e.g., in horizontal and vertical pixels, between the reference block(s) and the current block. For example, in a single reference mode, an inter-predicted block for a current block may be generated from a single reference block identified as the predictive block in a reference frame by one motion vector, while for a mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vector(s) may be coded and included in the bitstream in various manners.

[0155] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter prediction (in the decoding system or encoding system). In some implementations, the reference frames in the DPB may be tagged as either short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter prediction for blocks in the current frame or in a predefined number (e.g., two) subsequent video frames that are closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that may be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information regarding such tags for short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which is either numbered according to the playback sequence in an absolute manner, or associated with a group of pictures starting, for example, with an I-frame.

[0156] In some example implementations, one or more reference picture lists including identification information of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction, denoted as L0 reference (or reference list 0), and two picture reference lists may be formed for bidirectional inter prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. The unidirectional inter prediction may be in a single reference mode or may be in a mixed reference mode when multiple references for generation of a prediction block by weighted averaging in a mixed prediction mode are on the same side of the block to be predicted. Bidirectional inter prediction may be a mixed mode only in that bidirectional inter prediction involves at least two reference blocks.

[0157] Context Adaptive Arithmetic Coding Engine

[0158] FIG. 18 shows a flow chart for an example decoding of a single binary decision. Specifically, FIG. 18 shows an example for decoding a single binary decision in VVC. A context-adaptive arithmetic coding engine (CABAC) engine as used in HEVC may use a table-based probability transition process between 64 different representative probability states. In HEVC, the range ivlCurrRange may represent that the state of the coding engine is quantized to a set of four values ​​before the calculation of a new interval range. HEVC state transitions may be implemented using a table that contains all 64×4 8-bit pre-computed values ​​to approximate the value of ivlCurrRange*pLPS(pStateIdx), where pLPS is the probability of the least probable symbol (LPS) and pStateIdx is the index of the current state. The decoding decision may be implemented using a pre-computed LUT. The first ivlLpsRange is found using the LUT as follows: The ivlLpsRange is then used to update the ivlCurrRange and calculate the output binVal. An example formula is: ivlLpsRange=rangeTabLps[pStateIdx][qRangeIdx] (1) In VVC, the probability may be linearly represented by the probability index pStateIdx. Therefore, all calculations may be done using formulas without LUT operations. To improve the accuracy of the probability estimation, a multi-hypothesis probability update model may be applied. The pStateIdx used in the interval subdivision in the binary arithmetic coder is a combination of two probabilities pStateIdx0 and pStateIdx1. The two probabilities may be associated with each context model and are updated independently with different adaptation rates. The adaptation rates of pStateIdx0 and pStateIdx1 for each context model are pre-trained based on the statistics of the associated bin. The probability estimate pStateIdx is the average of the estimates from the two hypotheses.

[0159] Similar to HEVC, VVC CABAC may also have a QP-dependent initialization process that is invoked at the beginning of each slice. Given an initial value of the luma QP for a slice, the initial probability state of the context model, denoted as preCtxState, may be derived as follows: m=slopeIdx×5-45 (2) n=(offsetIdx<<3)+7 (3) preCtxState=Clip3(1,127,((m×(QP-32))>>4)+n) (4) Here, slopeIdx and offsetIdx are limited to 3 bits, and the entire initialization value is represented with 6-bit precision. The probability state preCtxState directly represents the probabilities in the linear domain. Thus, preCtxState only requires an appropriate shift operation before input to the arithmetic coding engine, saving the mapping from the logarithmic domain to the linear domain as well as a 256-byte table: pStateIdx0=preCtxState<<3 (5) pStateIdx1=preCtxState<<7 (6)

[0160] Multi-symbol arithmetic coding

[0161] In AV1, an M-ary arithmetic coding engine may be used to entropy code syntax elements. Each syntax element may be associated with an alphabet of M elements, where M can be any integer value between 2 and 16. The input to the coding is the M-ary symbols and a coding context consisting of a set of M probabilities represented by a cumulative distribution function (CDF). The probabilities are updated after encoding / parsing each syntax element. The CDF is an array of M 15-bit integers such that: C = [c0,c1,…,c (M-2) ,2 15 ] (7) where cn / 32768 is the probability that the symbol is less than or equal to n. The probability update is performed using the following formula: cm =c m (1-α) m∈[0,symbol) c m =c m +α·(1-c m ) m∈[symbol,M-1) (8) where α is a probability update rate that adapts based on the number of times a symbol is decoded (up to a maximum of 32), and m is the index of the element in the CDF. This adaptation of α may allow for faster probability updates at the beginning of encoding / parsing a syntax element. The M-ary arithmetic coding process follows a conventional arithmetic coding engine design, but only the most significant 9 bits of the 15-bit probability value are input to the arithmetic coder / decoder. The probability update rate α associated with a symbol is calculated based on the number of occurrences for the associated symbol when parsing the bitstream, and the value of α is reset at the beginning of a frame or tile using the following formula: α=1 / 2^(3+(count>15)+(count>32)+min(log2(M),2)) (9) From the above equation, the probability update rate has a larger value initially and saturates after 32 occurrences have occurred.

[0162] The multi-symbol arithmetic coding engine in AV1 may be a counter to update the window size, which may determine the cumulative distribution function (CDF) update rate of the context. After the context is updated 32 times, the window size may stabilize at 2^(5+min(log2(M),2)). However, for different areas of the picture, the objects may be different, and therefore the arithmetic coding engine may need to use a different window size or CDF or a combination of both for each context to better adapt the content as the coding proceeds from one object to a different object.

[0163] Block-level updates for arithmetic coding

[0164] Referring back to Figures 5-6, an example embodiment is shown in which arithmetic coding may be performed. The arithmetic coding may assume a fixed window size. The window size may be controlled by α as discussed below. For decoding, the frequency of updates may be modified. Different decoding may have different windows and / or window sizes. As more syntax is decoded, the value of α may change. A count may be recorded, and the value of α may be a function of the count as discussed below.

[0165] The probability update rate α is a function of the count and a value M, denoted as f(count,M). The count may be the number of occurrences for the associated syntax / context when parsing the bitstream, and M is the number of different symbol values ​​for the associated syntax / context. The value of the count may be restarted at a predefined value at the block level. In various embodiments, the probability update rate α may be restarted at the CU level, PU level, TU level, CTU level, or at a specified block size level. The block size for restarting the update rate α may be specified in a high level syntax, including, but not limited to, the VPS, PPS, SPS, APS, picture / frame header, slice header, tile header, or CTU header.

[0166] In some embodiments, the predefined values ​​for the count after restart include, but are not limited to, 0, 1, 2, ..., 32, ..., 128. nThere may be values ​​of n = n = n , where n is an integer. In one embodiment, the restart of the probability update rate α may be controlled differently for different syntaxes / contexts. In one embodiment, the restart of the probability update rate α is applied only to a selected group of syntaxes / contexts. In one embodiment, all syntaxes or contexts are grouped into different sets. Each set of syntaxes or contexts may restart the probability update rate α together. For different sets of syntaxes or contexts, the probability update rate α may be restarted at different times or with different values.

[0167] In some embodiments, there may be flags for syntax indication. For example, one flag may be signaled at the block level to indicate whether a selected group of syntax or context restarts the counter for updating the probability update rate α. In one embodiment, syntax or context are grouped into various sets, and each set of syntax or context restarts the probability update rate α that are jointly controlled by one flag.

[0168] When the probability value of a syntax / context is copied from another frame to the current frame, the value of the count may also be copied or restarted with a predefined value. In an embodiment, the predefined value may not be 0 and may be specified for each context. In an embodiment, the value of the count may be copied along with the probability value when the decoding / encoding is processing a syntax located at a predefined position in the picture or bitstream.

[0169] The probability or cumulative distribution function (CDF) associated with the syntax / context may be reinitialized with a predefined value at the block level 1. In an embodiment, the CDF is initialized at the CU level, PU level, TU level, or CTU level, or at a specified block size level. In an embodiment, the block size for reinitializing the CDF may be specified in a higher level syntax, including, but not limited to, the VPS, PPS, SPS, APS, picture / frame header, slice header, tile header, or CTU header.

[0170] In some embodiments, the predefined values ​​include, but are not limited to, values ​​corresponding to a uniform distribution. In some embodiments, the predefined values ​​include, but are not limited to, values ​​that are stored at some given instance during encoding or decoding.

[0171] In one embodiment, the reinitialization of the probability / CDF can be controlled differently for different syntaxes / contexts. Reinitialization can be applied only to selected groups of syntaxes or contexts. In one embodiment, all syntaxes or contexts are grouped into different sets, and each set of syntaxes or contexts reinitializes the CDF together. For different sets of syntaxes or contexts, the CDF can be reinitialized at different times or with different values.

[0172] In one embodiment, a flag can be signaled at the block level to indicate whether a selected group of syntaxes or contexts reinitialize the corresponding CDF. In one embodiment, all syntaxes or contexts are grouped into various sets, and each set of syntaxes or contexts reinitializes a CDF that is controlled together by one flag.

[0173] As mentioned above, there may be a block-level restart or reinitialization of the count using a predefined value. The counter is reset to this predefined value, which allows for content-based adaptation. The block-level options are described above.

[0174] 19A shows a flow chart of a method according to an example embodiment of the present disclosure. In block 1902, a count determination is made for the number of syntaxes / contexts when updating the window size. In block 1904, a probability update rate is calculated as a function of the count of the associated syntax and as a function of the number of distinct symbols. In block 1906, the count is updated at the block level with a predefined value as a result of the restart. In block 1908, the predefined value for the count is used for the next iteration / frame.

[0175] FIG. 19B illustrates a flowchart of a method according to another exemplary embodiment of the present disclosure. In block 1910, the number of occurrences of a context / syntax is counted as a value of a count. In block 1912, a cumulative distribution function (CDF) update rate is determined for the context. This determination is based on the count and may be based on a number of distinct symbols. In block 1914, the CDF is reinitialized at the block level with a predefined value for the count. This may be iteratively repeated using the count, now with a predefined value for the count.

[0176] Although described as a reference frame, it may also include a reference frame pair. In one embodiment, if multiple motion vectors point to one reference frame or one reference frame pair, the smallest difference value referenced by these motion vectors may be marked as the score value of one reference frame or one reference frame pair. In another embodiment, if there are no motion vectors pointing to a reference frame or a reference frame pair, the score value of that reference frame or reference frame pair is marked as the maximum allowed value. In another embodiment, if multiple reference frames or multiple reference frame pairs have the same score value, the ranking order of these reference frames or reference frame pairs is the same as the scanning order of the spatial reference motion information and / or the temporal reference motion information. In another embodiment, if multiple reference frames or multiple reference frame pairs have the same score value, the ranking order of these reference frames or reference frame pairs depends on the frequency of occurrence of these reference frames or reference frame pairs used in the spatial reference motion information and / or the temporal reference motion information.

[0177] In one embodiment, all allowed unidirectional and bidirectional composite reference frame pairs are ranked together by using a TM method, and the index of the reference frame pair for the current block in this ranking order is signaled in the bitstream. In one embodiment, all allowed single reference frames are ranked together by using a TM method, and the index of the reference frame for the current block in this ranking order is signaled in the bitstream.

[0178] In an embodiment, a method for reordering reference frames for each block by template matching (TM) includes: comparing a template of a current block and a template of a reference block for motion information, the motion information including spatial reference motion information or temporal reference motion information; calculating a difference between the template of the current block and the template of the reference block; determining a score value for the associated reference frame based on the calculated difference; and reordering the reference frames based on the determined score. The reference frames further include a reference frame pair. The TM includes a decoder-side motion vector derivation for refining the motion information of the current block. The reordering further includes ranking the available reference frames based on the score value for each of the reference frames. If the score values ​​of the multiple reference frames are equivalent, the ranking corresponds to a scanning order of the spatial reference motion information or the temporal reference motion information. If the score values ​​of the multiple reference frames are equivalent, the ranking is based on a frequency of occurrence of these reference frames used in the spatial reference motion information or the temporal reference motion information. The calculating includes at least one of sum of absolute differences (SAD), sum of squared differences (SSD), mean squared error (MSE), or sum of absolute transformed differences (SATD). The template includes a top or left neighboring block. The spatial reference motion information includes one or more spatial motion vectors. The temporal reference motion information includes one or more temporal motion vectors. When multiple motion vectors point to one of the reference frames, one of the motion vectors with the smallest difference is used to determine the score value. When there is no motion vector that points to one of the reference frames, the score value is determined to be the maximum allowed value. The allowed unidirectional and bidirectional composite reference frames are ranked together by using TM, and the index of the reference frame for the current block is signaled in the bitstream. The allowed single reference frames are ranked together by using TM, and the index of the reference frame for the current block is signaled in the bitstream.

[0179] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The term block may include a prediction block, a coding block, or a coding unit, i.e., a CU. The term block may also be used to refer to a transform block. The block size may refer to either the width or height of the block, or the maximum value of the width and height, or the minimum value of the width and height, or the area size (width*height), or the aspect ratio (width:height or height:width).

[0180] The techniques described above can be implemented as computer software using computer readable instructions and can be physically stored on one or more computer readable media. For example, Figure 20 illustrates a computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter.

[0181] Computer software may be coded using any suitable machine code or computer language and may be subject to assembly, compilation, linking, or similar mechanisms to produce code containing instructions that are executable by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly, or through interpretation, microcode execution, etc.

[0182] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0183] The components illustrated in Figure 20 for the computer system (2000) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. Neither the arrangement of components should be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (2000).

[0184] The computer system (2000) may include certain human interface input devices that may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0185] The input human interface devices may include one or more (only one of each is shown) of a keyboard (2001), a mouse (2002), a trackpad (2003), a touch screen (2010), a data glove (not shown), a joystick (2005), a microphone (2006), a scanner (2007), and a camera (2008).

[0186] The computer system (2000) may also include some type of human interface output device. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2010), data gloves (not shown), or joystick (2005); although there may be haptic feedback devices that do not act as input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens; each may or may not have touch screen input capability, each may or may not have haptic feedback capability, some of which may output two-dimensional visual output or higher than three-dimensional output through such means as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0187] The computer system (2000) may also include human accessible storage and associated media, such as optical media including CD / DVD ROM / RW (2020) along with CD / DVD or similar media (2021), thumb drives (2022), removable hard drives or solid state drives (2023), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, etc.

[0188] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0189] The computer system (2000) may also include an interface (2054) to one or more communication networks (2055). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, in-vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable television, satellite television, terrestrial broadcast television, in-vehicle and industrial including CANBus, etc. Some networks typically require an external network interface adapter that is attached to some kind of general purpose data port or peripheral bus (2049) (e.g., a USB port of the computer system (2000)). Others are typically integrated into the core of the computer system (2000) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), unidirectional transmit only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide area digital networks. With each of these networks and network interfaces as described above, certain protocols and protocol stacks may be used.

[0190] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2040) of the computer system (2000).

[0191] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2043), hardware accelerators for certain tasks (2044), graphics adapters (2050), and the like. These devices may be connected through a system bus (2048), along with read-only memory (ROM) (2045), random access memory (2046), and internal mass storage devices (2047), such as internal non-user-accessible hard drives, solid-state drives (SSDs), and the like. In some computer systems, the system bus (2048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (2048) or through a peripheral bus (2049). In one example, a screen 2010 may be connected to a graphics adapter 2050. Architectures for peripheral buses include PCI, USB, and the like.

[0192] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute certain instructions that may combine to constitute the computer code described above. The computer code may be stored in a ROM (2045) or a RAM (2046). Temporary data may also be stored in the RAM (2046), while persistent data may be stored, for example, in an internal mass storage device (2047). Rapid storage and retrieval from any of the memory devices may be enabled through the use of a cache memory that may be closely associated with one or more of the CPU (2041), GPU (2042), mass storage device (2047), ROM (2045), RAM (2046), and the like.

[0193] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0194] As a non-limiting example, a computer system having the architecture (2000), and specifically the core (2040), can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer readable media. Such computer readable media can be user accessible mass storage as introduced above as well as media associated with some type of storage of the core (2040) of a non-transitory nature such as mass storage (2047) or ROM (2045) internal to the core. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2040). The computer readable media can include one or more memory devices or chips depending on the particular needs. The software can cause the core (2040) and specifically the processor therein (including CPU, GPU, FPGA, etc.) to execute certain processes or certain specific portions described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to the processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2044)), which may operate in place of or together with software to perform particular processes or portions of particular processes described herein. Reference to software includes logic, and vice versa, as appropriate. Reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0195] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents, which fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within the spirit and scope of the present disclosure.

[0196] Appendix A: Acronyms ALF: Adaptive Loop Filter AMVP: Advanced Motion Vector Prediction APS: Adaptation Parameter Set ASIC: Application-Specific Integrated Circuit AV1: AOMedia Video 1 AV2: AOMedia Video 2 BCW: Bi-prediction with CU-level Weights BM: Bilateral Matching BMS: benchmark set CANBus: Controller Area Network Bus CC-ALF: Cross-Component Adaptive Loop Filter CCSO: Cross-Component Sample Offset CD: Compact Disc CDEF: Constrained Directional Enhancement Filter CDF: Cumulative Density Function CfL: Chroma from Luma CIIP: Combined intra-inter prediction CPU: Central Processing Unit CRT: Cathode Ray Tube CTB: Coding Tree Block CTU: Coding Tree Unit CTU: Coding Tree Unit CU: Coding Unit DMVR: Decoder-side Motion Vector Refinement DPB: Decoded Picture Buffer DPS: Decoding Parameter Set DVD: Digital Video Disc FPGA: Field Programmable Gate Areas GBI: Generalized Bi-prediction GOP: Group of Pictures GPU: Graphics Processing Unit GSM: Global System for Mobile communications HDR: high dynamic range HEVC: High Efficiency Video Coding HRD: Hypothetical Reference Decoder IBC (or IntraBC): Intra-Block Copy IC: Integrated Circuit ISP: Intra Sub-Partition JEM: joint exploration model JVET: Joint Video Exploration Team LAN: Local Area Network LCD: Liquid-Crystal Display LR: Loop Restoration Filter LSO: Local Sample Offset LTE: Long-Term Evolution MMVD: Merge Mode with Motion Vector Difference MPM: most probable mode MV: Motion Vector MVD: Motion Vector Difference MVP: Motion Vector Predictor OLED: Organic Light-Emitting Diode PB: Prediction Block PCI: Peripheral Component Interconnect PDPC: Position Dependent Prediction Combination PLD: Programmable Logic Device POC: Picture Order Count PPS: Picture Parameter Set PU: Prediction Unit PU: Prediction Unit RAM: Random Access Memory ROM: Read-Only Memory RPS: Reference Picture Set SAD: Sum of Absolute Difference SAO: Sample Adaptive Offset SB: Super Block SCC: Screen Content Coding SDP: Semi Decoupled Partitioning SDR: standard dynamic range SDT: Semi Decoupled Tree SEI: Supplementary Enhancement Information SNR: Signal Noise Ratio SPS: Sequence Parameter Setting SSD: solid-state drive SST: Semi Separate Tree TM: Template Matching TU: Transform Unit TU: Transform Unit USB: Universal Serial Bus VPS: Video Parameter Set VUI: Video Usability Information VVC: versatile video coding WAIP: Wide-Angle Intra Prediction

Claims

1. 1. A processor-implemented method for updating a window size when decoding a video stream, the method comprising: receiving an encoded video stream including a number of syntaxes and a number of symbols associated with the syntaxes; determining a count for said number of syntax elements when updating a window size; determining a number of symbols associated with the syntax; calculating a probability update rate as a function of determined counts for associated syntaxes and determined numbers of symbols associated with those syntaxes, the values ​​of the counts having predefined values ​​at a block level; and updating a context corresponding to the associated syntax with the calculated probability update rate. method.

2. restarting the value of the count with the predefined value at a block level. The method of claim 1.

3. copying the syntax probability values ​​from another frame to the current frame; and restarting each value of the count for each frame with the predefined value. The method of claim 2.

4. 3. The method of claim 2, wherein the restart is performed at a coding unit (CU) level, a prediction unit (PU) level, a transform unit (TU) level, a coding tree unit (CTU) level, or a specified block size level.

5. 3. The method of claim 2, wherein the block size for the restart is specified in a high level syntax including a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter set (SPS), an adaptation parameter set (APS), a picture header, a frame header, a slice header, a tile header, or a coding tree unit (CTU) header.

6. The predefined value of the count after the restart is 2 n 3. The method of claim 2, comprising the value: where n is an integer.

7. The method of claim 2 , wherein the restart is syntax dependent and the probability update rate is controlled differently for different syntaxes.

8. 8. The method of claim 7, wherein the probability update rate applies to a particular syntax group, such that the probability update rate is restarted at different times depending on the syntax group, and the probability update rate is restarted at different values ​​depending on the syntax group.

9. 8. The method of claim 7, wherein a flag is signaled at a block level to indicate whether a selected syntax group restarts the value of the count to update the probability update rate.

Images