Context-based adaptive binary arithmetic coding (CABAC) context modeling with information from temporal neighbors

In video encoding and decoding, the CABAC context model combines the information of time-horizontal blocks and spatial neighbor blocks, optimizes the intra-frame and inter-frame prediction of video, solves the problem of low video compression efficiency in the prior art, and achieves the improvement Efficient video data transmission.

CN118975227APending Publication Date: 2024-11-15TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380014668.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-28
Filing Date
2023-06-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When existing video encoding and decoding technologies utilize time redundancy and spatial redundancy, it is difficult to effectively utilize the information of the time homologous block and spatial adjacent block of the current block, resulting in low video compression efficiency.

Method used

The context model of the syntax element is determined based on the CABAC context model and combined with the information of the temporal and spatial neighbor blocks of the current block during the video encoding and decoding process to optimize the intra- and inter-frame prediction of the video.

Benefits of technology

It improves the efficiency of video encoding and decoding, reduces the amount of data, maintains video quality, and is suitable for video transmission in different devices and network environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975227A_ABST
    Figure CN118975227A_ABST
Patent Text Reader

Abstract

Aspects of the present disclosure provide methods and apparatus for video encoding / decoding. In some examples, an apparatus for video decoding includes processing circuitry. Processing circuitry receives coded information for a current block in a current picture from a coded video bitstream. The coded information includes a syntax element associated with the current block, the syntax element indicating a decoding parameter of the current block. The processing circuitry determines a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element based on at least a temporally collocated block of the current block. The temporal parity block is in a picture different from the current picture. Further, the processing circuitry decodes the syntax elements based on the CABAC context model, and reconstructs the current block based on the syntax elements decoded based on the CABAC context model.
Need to check novelty before this filing date? Find Prior Art

Description

Cross-references

[0001] This application claims the benefit of priority to U.S. Patent Application No. 18 / 215,244, filed on June 28, 2023, for “CONTEXT-BASED ADAPTIVE BINARY ARITHMETIC CODING (CABAC) CONTEXT MODELING WITH INFORMATION FROM TEMPORAL NEIGHBORS,” which claims the benefit of priority to U.S. Provisional Application No. 63 / 452,379, filed on March 15, 2023, for “Context-Based Adaptive Binary Arithmetic Coding (CABAC) Context Modeling with Information from Spatial Neighbors.” The disclosure of the prior application is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure describes implementations that are generally related to video coding. Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent that work is described in this background section, the work of the presently named inventors and aspects of the description that may not otherwise be qualified as prior art at the time of filing are neither explicitly nor implicitly admitted as prior art against the present disclosure.

[0004] Image / video compression can help transfer image / video files between different devices, storage devices, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial redundancy and temporal redundancy. In an example, a video codec can use a technique called intra-frame prediction, which can compress images based on spatial redundancy. For example, intra-frame prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-frame prediction, which can compress images based on temporal redundancy. For example, inter-frame prediction can use motion compensation to predict samples in the current picture based on previously reconstructed pictures. Motion compensation is usually indicated by a motion vector (MV). Summary of the invention

[0005] Various aspects of the present disclosure provide methods and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a receiving circuit system and a processing circuit system. The processing circuit system receives decoded information of a current block in a current picture from a decoded video bitstream. The decoded information includes a syntax element associated with the current block, which indicates a decoding parameter of the current block. The processing circuit system determines a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element based on at least a temporal co-located block of the current block. The temporal co-located block is in a picture different from the current picture. In addition, the processing circuit system decodes the syntax element based on the CABAC context model, and reconstructs the current block based on the syntax element decoded based on the CABAC context model.

[0006] According to one aspect of the present disclosure, the temporal co-located block is located in a reference picture of the current picture at the same coordinates as the current block. In some examples, the processing circuit system determines the CABAC context model based on one or more sub-blocks located at predefined positions relative to the temporal co-located block. For example, the processing circuit system determines the CABAC context model based on at least a first sub-block at a middle position relative to the temporal co-located block and a second sub-block at a lower right position relative to the temporal co-located block.

[0007] In some examples, the processing circuit system determines a CABAC context model associated with the syntax element based on at least spatial neighboring blocks and at least temporal co-located blocks of the current block. In some embodiments, the processing circuit system scans at least spatial neighboring blocks and at least temporal co-located blocks to determine the number of blocks that meet the requirements in at least spatial neighboring blocks and at least temporal co-located blocks, and determines the CABAC context model based on the number of blocks that meet the requirements. In an example, the syntax element is a flag indicating whether the current block is only intra-coded, and the processing circuit system scans at least spatial neighboring blocks and at least temporal co-located blocks to determine the number of blocks that are intra-coded in at least spatial neighboring blocks and at least temporal co-located blocks, and determines the CABAC context model for deriving the flag based on the number of blocks.

[0008] It should be noted that the syntax element can be one of the following: inter-frame affine flag, merge sub-block flag, local illumination compensation flag, non-inter-frame flag, decoding unit skip flag, prediction mode flag or inter-frame matrix-based intra-prediction (mip) flag.

[0009] Various aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding / encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Additional features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0011] Figure 1 is a schematic illustration of an exemplary block diagram of a communication system (100).

[0012] Figure 2 is a schematic illustration of an exemplary block diagram of a decoder.

[0013] Figure 3 is a schematic illustration of an exemplary block diagram of an encoder.

[0014] Figure 4 A table showing quantized values ​​for the least probable symbol range in some examples.

[0015] Figure 5 A flowchart outlining the process of decoding a single binary decision in some examples is shown.

[0016] Figure 6 Some example tables are shown.

[0017] Figure 7 Another table is shown in some examples.

[0018] Figure 8 Diagram showing the locations of spatial neighbors in some examples.

[0019] Fig. 9 Diagram showing the locations of spatial neighbors in some examples.

[0020] Fig.10 A diagram showing time-colocated blocks in some examples.

[0021] Fig.11 A flowchart outlining a decoding process according to some embodiments of the present disclosure is shown.

[0022] Fig.12 A flowchart outlining an encoding process according to some embodiments of the present disclosure is shown.

[0023] Fig.13 is a schematic illustration of an exemplary computer system according to an embodiment. DETAILED DESCRIPTION

[0024] Figure 1 A block diagram of a video processing system (100) in some examples is shown. The video processing system (100) is an example of an application of the disclosed subject matter, a video encoder and a video decoder in a streaming environment. The disclosed subject matter can be equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0025] The video processing system (100) includes a capture subsystem (113) that may include a video source (101), such as a digital camera, that creates, for example, an uncompressed video picture stream (102). In an example, the video picture stream (102) includes samples captured by the digital camera. The video picture stream (102) is depicted as a thick line to emphasize the high amount of data when compared to the encoded video data (104) (or decoded video bitstream), and the video picture stream (102) may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination of hardware and software to implement or embody aspects of the disclosed subject matter as described in more detail below. The encoded video data (104) (or decoded video bitstream) is depicted as a thin line to emphasize the lower data volume when compared to the video picture stream (102), and the encoded video data (104) can be stored on the streaming server (105) for future use. One or more streaming client subsystems such as Figure 1 The client subsystems (106) and (108) in the streaming server (105) can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an outgoing video picture stream (111) that can be presented on a display (112) (e.g., a display screen) or another presentation device (not depicted). In some streaming systems, the encoded video data (104), (107) and (109) (e.g., a video bitstream) can be encoded according to certain video decoding standards / video compression standards. Examples of such standards include ITU-T Recommendation H.265. In examples, the video decoding standard under development is informally referred to as Versatile Video Coding (VVC). The disclosed subject matter can be used in the context of VVC.

[0026] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0027] Figure 2 An exemplary block diagram of a video decoder (210) is shown. The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., receiving circuitry). The video decoder (210) may replace Figure 1 The video decoder (110) in the example uses.

[0028] The receiver (231) may receive one or more decoded video sequences to be decoded by the video decoder (210). In an embodiment, the decoded video sequences are received one at a time, wherein the decoding of each decoded video sequence is independent of the decoding of other decoded video sequences. The decoded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (231) may receive the encoded video data with other data, such as decoded audio data and / or auxiliary data streams, which may be forwarded to their respective consuming entities (not depicted). The receiver (231) may separate the decoded video sequence from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter referred to as "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory may be external to the video decoder (210) (not depicted). In still other applications, there may be a buffer memory (not depicted) external to the video decoder (210) to, for example, prevent network jitter, and there may be another buffer memory (215) internal to the video decoder (210) to, for example, handle playback timing. When the receiver (231) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (215) may not be required, or the buffer memory (215) may be smaller. For use on an optimal packet network such as the Internet, the buffer memory (215) may be required, the buffer memory (215) may be relatively large and may advantageously have an adaptive size, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (210).

[0029] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the decoded video sequence. The categories of these symbols include: information for managing the operation of the video decoder (210), and possibly information for controlling a presentation device such as a presentation device (212) (e.g., a display screen), which is not part of the electronic device (230) but can be coupled to the electronic device (230), such as Figure 2 . The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser (220) may parse / entropy decode the received decoded video sequence. The decoding of the decoded video sequence may be performed according to a video decoding technique or a video decoding standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a subgroup parameter set for at least one subgroup of the subgroups of pixels in the video decoder from the decoded video sequence based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the decoded video sequence.

[0030] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0031] Depending on the type of decoded video picture or part of a decoded video picture (e.g., inter-frame and intra-frame pictures, inter-frame blocks and intra-frame blocks) and other factors, the reconstruction of the symbol (221) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the decoded video sequence by the parser (220). For the sake of simplicity, such subgroup control information flow between the parser (220) and the following multiple units is not depicted.

[0032] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated into each other. However, for the purposes of describing the disclosed subject matter, the conceptual subdivision into the following functional units is appropriate.

[0033] The first unit is a sealer / inverse transform unit (251). The sealer / inverse transform unit (251) receives quantized transform coefficients as symbols (221) from the parser (220) and control information including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit (251) can output blocks including sample values ​​which can be input into an aggregator (255).

[0034] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to an intra-coded block. An intra-coded block may be a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of a current picture. Such predictive information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the reconstructed block using surrounding already reconstructed information extracted from a current picture buffer (258). For example, the current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (255) adds the prediction information already generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) on a per-sample basis.

[0035] In other cases, the output samples of the scaler / inverse transform unit (251) may belong to an inter-frame decoded and possibly motion compensated block. In this case, the motion compensated prediction unit (253) may access the reference picture memory (257) to extract samples for prediction. After the extracted samples are motion compensated according to the symbols (221) belonging to the block, these samples may be added to the output of the scaler / inverse transform unit (251) (in this case, called residual samples or residual signals) by the aggregator (255) to generate output sample information. The address within the reference picture memory (257) from which the motion compensated prediction unit (253) extracts the prediction samples may be controlled by a motion vector, which is provided to the motion compensated prediction unit (253) in the form of a symbol (221), which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (257) when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0036] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in a loop filter unit (256). The video compression techniques may include in-loop filter techniques controlled by parameters included in a decoded video sequence (also referred to as a decoded video bitstream) which may be obtained by the loop filter unit (256) as symbols (221) from the parser (220). The video compression may also be responsive to meta-information obtained during decoding of a decoded picture or a previous (in decoding order) portion of a decoded video sequence, as well as to previously reconstructed and loop filtered sample values.

[0037] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and stored in a reference picture memory (257) for use in future inter-picture prediction.

[0038] Certain decoded pictures, once fully reconstructed, may be used as reference pictures for future prediction. For example, once a decoded picture corresponding to a current picture is fully reconstructed and the decoded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257) and a new current picture buffer may be reallocated before starting to reconstruct a subsequent decoded picture.

[0039] The video decoder (210) may perform decoding operations according to a standard or predetermined video compression technology such as ITU-T H.265 Recommendation. The decoded video sequence may conform to the syntax specified by the video compression technology or standard used in the sense that the decoded video sequence follows both the syntax of the video compression technology or standard and the profile recorded in the video compression technology or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technology or standard as tools that can only be used under the profile. For compliance, the complexity of the decoded video sequence is also required to be within the range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the hypothetical reference decoder (HRD) specification and the metadata of the HRD buffer management signaled in the decoded video sequence.

[0040] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the decoded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0041] Figure 3 An exemplary block diagram of a video encoder (303) is shown. The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit system). The video encoder (303) may replace Figure 1 The video encoder (103) in the example uses.

[0042] The video encoder (303) can be used to capture video images to be decoded by the video encoder (303) from a video source (301) (not Figure 3 In another example, the video source (301) is a part of the electronic device (320).

[0043] The video source (301) may provide a source video sequence in the form of a digital video sample stream to be decoded by the video encoder (303), the digital video sample stream may have any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (301) may be a storage device storing previously prepared videos. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that are given motion when viewed sequentially. The pictures themselves may be organized as a spatial pixel array, wherein each pixel may include one or more samples, depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0044] According to an embodiment, the video encoder (303) can decode and compress the pictures of the source video sequence into a decoded video sequence (343) in real time or under any other time constraints required. Implementing the appropriate decoding speed is a function of the controller (350). In some embodiments, the controller (350) controls other functional units as described below and is functionally coupled to the other functional units. The coupling is not depicted for clarity. The parameters set by the controller (350) may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured to have other suitable functions related to the video encoder (303) optimized for certain system designs.

[0045] In some embodiments, the video encoder (303) is configured to operate in a decoding loop. As an oversimplified description, in an example, the decoding loop may include a source decoder (330) (e.g., responsible for creating symbols such as a symbol stream based on an input picture to be decoded and a reference picture) and a (local) decoder (333) embedded in the video decoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to the way the (remote) decoder creates sample data. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local encoder and the remote encoder. In other words, the prediction part of the encoder will "see" as a reference picture sample the same sample value as the sample value that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift caused when synchronization cannot be maintained, for example due to channel errors) is also used in some related technologies.

[0046] The operation of the "local" decoder (333) can be combined with the above Figure 2 The operation of the "remote" decoder described in detail is identical to that of the video decoder (210). However, reference is also briefly made to Figure 2 , since the symbols are available and the encoding / decoding of the symbols to the decoded video sequence by the entropy decoder (345) and the parser (220) can be lossless, the entropy decoding portion of the video decoder (210) including the buffer memory (215) and the parser (220) may not be fully implemented in the local decoder (333).

[0047] In an embodiment, decoder techniques other than parsing / entropy decoding present in a decoder are present in a corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on decoder operation. Since encoder techniques are opposite to the fully described decoder techniques, the description of encoder techniques can be simplified. A more detailed description is provided below in certain parts.

[0048] During operation, in some examples, the source decoder (330) may perform motion compensated predictive coding, which predictively codes an input picture with reference to one or more previously decoded pictures from a video sequence designated as "reference pictures." In this manner, the decoding engine (332) decodes the differences between pixel blocks of the input picture and pixel blocks of a reference picture, which may be selected as a prediction reference for the input picture.

[0049] The local video decoder (333) can decode the decoded video data of the picture that can be designated as the reference picture based on the symbols created by the source decoder (330). The operation of the decoding engine (332) can advantageously be a lossy process. When the decoded video data can be decoded at the video decoder ( Figure 3 When the video encoder (303) is decoded at a remote location (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) replicates the decoding process that may be performed by the video decoder on the reference picture, and may cause the reconstructed reference picture to be stored in the reference picture memory (334). In this way, the video encoder (303) may locally store a copy of the reconstructed reference picture that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the remote video decoder.

[0050] The predictor (335) may perform a prediction search for the decoding engine (332). That is, for a new picture to be decoded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may be used as appropriate prediction references for the new picture. The predictor (335) may operate on a pixel block by pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references taken from multiple reference pictures stored in the reference picture memory (334).

[0051] The controller (350) may manage decoding operations of the source decoder (330), including, for example, the setting of parameters and subgroup parameters for encoding video data.

[0052] The outputs of all the above functional units may be entropy decoded in an entropy decoder (345). The entropy decoder (345) converts the symbols generated by the various functional units into a decoded video sequence by applying lossless compression according to techniques such as Huffman decoding, variable length decoding, arithmetic decoding, etc.

[0053] The transmitter (340) can buffer the decoded video sequence created by the entropy decoder (345) in preparation for transmission via the communication channel (360), which can be a hardware / software link to a storage device storing the encoded video data. The transmitter (340) can combine the decoded video data from the video encoder (303) with other data to be transmitted, such as decoded audio data and / or auxiliary data streams (source not shown).

[0054] The controller (350) may manage the operation of the video encoder (303). During decoding, the controller (350) may assign a decoded picture type to each decoded picture, which may affect the decoding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following picture types:

[0055] An intra picture (I picture) may be a picture that can be coded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of those variations of I pictures and their corresponding applications and features.

[0056] A predictive picture (P picture) may be a picture that may be coded and decoded using inter prediction or intra prediction that uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0057] Bi-directional predictive pictures (B pictures), which can be pictures that can be coded and decoded using inter-prediction or intra-prediction using up to two motion vectors and reference indices to predict sample values ​​for each block. Similarly, multi-predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0058] The source picture may typically be spatially subdivided into a plurality of blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples, respectively), and coded block by block. These blocks may be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments applied to the corresponding pictures of the blocks. For example, blocks of an I picture may be non-predictively coded, or may be predictively coded (spatial prediction or intra prediction) with reference to already coded blocks of the same picture. Pixel blocks of a P picture may be predictively coded with reference to one previously decoded reference picture via spatial prediction or via temporal prediction. Blocks of a B picture may be predictively coded with reference to one or two previously decoded reference pictures via spatial prediction or via temporal prediction.

[0059] The video encoder (303) may perform decoding operations according to a predetermined video decoding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (303) may perform various compression operations, including predictive decoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the decoded video data may conform to the syntax specified by the video decoding technique or standard being used.

[0060] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source decoder (330) may include such data as part of the decoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0061] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often referred to simply as intra-picture prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a particular picture in encoding / decoding, which is referred to as the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously decoded and is still buffered in the video, the block in the current picture can be decoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case of using multiple reference pictures, the motion vector may have a third dimension that identifies the reference picture.

[0062] In some embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past and in the future in display order, respectively). A block in the current picture may be decoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0063] In addition, merge mode technology can be used in inter-picture prediction to improve decoding efficiency.

[0064] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, a picture in a video picture sequence is partitioned into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), each of which is a luminance CTB and two chrominance CTBs. Each CTU can be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be divided into a 64×64 pixel CU, or 4 32×32 pixel CUs, or 16 16×16 pixel CUs. In the example, each CU is analyzed to determine the prediction type of the CU, such as an inter-prediction type or an intra-prediction type. Depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, the prediction operation in decoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0065] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In an embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0066] Some aspects of the present disclosure provide techniques for entropy coding, such as techniques for context-based adaptive binary arithmetic coding (CABAC). For example, a context model used in CABAC for coding a current block may be determined based on information from temporal neighboring blocks (also known as temporal co-located blocks) of the current block and / or spatial neighboring blocks of the current block. CABAC may be used to code various syntax elements, such as an inter-frame affine flag (e.g., inter_affine_flag), a sub-block merge flag (e.g., merge_subblock_flag), a local illumination compensation (LIC) flag (e.g., lic_flag), and the like. The inter-frame affine flag associated with a coding block is used to indicate whether the coding block is coded using affine motion compensated prediction. The sub-block merge flag associated with a coding block is used to indicate whether the coding block is coded using a sub-block motion compensation mode. The LIC flag associated with a coding block is used to indicate whether the coding block is coded using local illumination compensation.

[0067] CABAC is a decoding technique used in entropy decoding. Generally, the encoding process of CABAC includes a binarization step, a context modeling step, and an arithmetic decoding step.

[0068] In the binarization step of the CABAC-based encoding process, non-binary valued syntax elements may be mapped to binary sequences (also referred to as binary strings). When syntax elements have values ​​in binary form (eg, binary sequences), the binarization step may be bypassed.

[0069] In the context modeling step of the CABAC-based coding process, the probability model is determined depending on the previously encoded syntax elements. In some examples, the probability model (also referred to as context or context model in CABAC) can be represented by a probability state (also referred to as context state in some examples) and a most probable symbol (MPS) value. The probability state is associated with a probability value, and can implicitly indicate that the probability that a particular symbol (e.g., binary) is a least probable symbol (LPS) is equal to the probability value. The symbol can be an LPS or an MPS. For binary symbols, MPS and LPS can be 0 or 1. For example, if LPS is 1, MPS is 0; and if LPS is 0, MPS is 1. The probability value of the corresponding context is estimated, and the probability value can be used to entropy decode the symbol using an arithmetic decoder.

[0070] The arithmetic decoding step of the CABAC-based encoding process is based on the principle of recursive interval subdivision according to a probability model. In some examples, the arithmetic decoding step is processed by a state machine with a range parameter and a lower limit parameter (low parameter). The state machine can change the values ​​of the range parameter and the lower limit parameter based on the context (probability model) and the decoded binary sequence. The value of the range parameter indicates the size of the current range into which the decoded value (binary decoded value) falls, and the value of the lower limit parameter indicates the lower boundary of the current range. In the example, according to the probability state (e.g., associated with the probability value), the current range (CurrRange) is divided into a first sub-range (MpsRange) (also referred to as the MPS range of the current state) and a second sub-range (LpsRange) (also referred to as the LPS range of the current state). In the example, the second sub-range can be calculated by multiplication, for example, using equation (1): LpsRange = CurrRange × ρ Equation (1) Where ρ is the probability value that the current binary is LPS. The probability that the current binary is MPS can be calculated by (1-ρ). The first sub-range can be calculated by equation (2): MpsRange = CurrRange - LpsRange Equation (2)

[0071] In the example, when the current binary is MPS, the value of the lower limit parameter is maintained and the value of the range parameter is updated to MpsRange; and when the current binary is LPS, the value of the lower limit parameter is updated to (low+MpsRange), and the value of the range parameter is updated to LpsRange. Then, the CABAC encoding process can continue to the next binary in the binary sequence.

[0072] In some examples (e.g., HEVC), the value of the range parameter is expressed with 9 bits and the value of the floor parameter is expressed with 10 bits. In addition, a renormalization process may be performed to maintain the value of the range and the value of the floor with sufficient accuracy. For example, renormalization may be performed as long as the value of the range parameter is less than 256. Thus, after renormalization, the range parameter is equal to or greater than 256.

[0073] In some examples (e.g., HEVC), 64 possible probability values ​​of LPS may be used, and each MPS may be 0 or 1. In an example, the probability model may be stored as a 7-bit entry corresponding to 64 probability values ​​(64 probability states) and 2 possible values ​​of MPS (0 or 1). In each of the 7-bit entries, 6 bits may be allocated to represent the probability state, and 1 bit may be allocated for the MPS.

[0074] In some examples, to reduce the computations to derive the LPS range (e.g., the multiplications in equation (1)), the results for all cases are pre-computed, quantized, and stored as approximations in a lookup table. Thus, the LPS range can be obtained using a table lookup without any multiplication operations. For some devices or applications, it may be important to avoid multiplications to reduce computation and latency.

[0075] Figure 4 A table (400) of quantized values ​​of LPS ranges in some examples is shown. In some examples, the range parameter is expressed in 9 bits and the value of the range parameter is equal to or greater than 256. The ranges can be divided into four segments, for example referred to as segment 0, segment 1, segment 2, and segment 3. Segment 0 includes 64 ranges with values ​​from 256 to 319; segment 1 includes 64 ranges with values ​​from 320 to 383; segment 2 includes 64 ranges with values ​​from 384 to 447; segment 3 includes 64 ranges with values ​​from 448 to 511. In an example, the index of the segment can be derived using (range >> 6) & 3, where the range represents the range parameter expressed in 9 bits. In some examples, the range parameter is expressed in 9 bits, then the quantized value of the LPS range can be expressed using 8 bits, so the values ​​in table (400) can be expressed using 8 bits.

[0076] exist Figure 4 In the example of , the table (400) includes 4 columns corresponding to 4 segments of the range, respectively, and includes 64 rows corresponding to 64 probability states (associated with corresponding probability values), respectively. The value stored at the entry with the row index and the column index is the quantized value of the LPS range associated with the probability state (corresponding to the row index) and the range segment (corresponding to the column index). Figure 4 In the example of , the LPS range of the probability state is quantized into four values ​​(ie, one value for each segment).

[0077] In some examples, a CABAC engine for performing CABAC-based encoding or decoding (e.g., in a video encoder or video decoder) may include a table (400) storing 64×4 8-bit values ​​to approximate the calculation in equation (1). Table (400) can be used to perform a table-based probability conversion process between 64 different probability states. For example, for a current range, a probability state is determined. The current range is used to determine a range segment indicating a column index in table (400). The probability state is used to determine a row index in table (400). A table lookup is then performed on table (400) to obtain a value stored in an entry having a row index and a column index, and the value is the approximate LPS range.

[0078] In some examples (e.g., VVC), the probability value of a probability state is linearly expressed by a probability index (represented by pStateIdx) and can be calculated using an equation without LUT operations. In order to improve the accuracy of the probability estimate, a multi-hypothesis probability update model can be applied. For example, the pStateIdx used in the interval subdivision in the binary arithmetic decoder is a combination of two probabilities pStateIdx0 and pStateIdx1 (in the case of two hypotheses). These two probabilities are associated with their respective context models and are updated independently at different adaptation rates. The adaptation rates of pStateIdx0 and pStateIdx1 of each context model can be pre-trained based on the statistics of the associated binary. In some examples, the probability estimate pStateIdx is the average of the estimates from the two hypotheses.

[0079] Figure 5A flowchart is shown outlining a process (500) for decoding a single binary decision in some examples (e.g., VVC). In some examples, a context table (e.g., represented by a variable ctxTable) and a context index (e.g., represented by ctxIdx) may be provided to the process (500) to obtain probability state information, such as two probabilities pStateIdx0 and pStateIdx1. The process (500) may operate on variables ivlCurrRange (indicating a current range) and ivlOffset (indicating a lower boundary of the current range) based on the probability state information, such as the two probabilities pStateIdx0 and pStateIdx1, and may output a decoded value binVal and update the variables ivlCurrRange and ivlOffset and the two probabilities pStateIdx0 and pStateIdx1. It should be noted that in some examples, the variables ivlCurrRange and ivlOffset are defined in a way that ivlOffset is greater than or equal to ivlCurrRange. The process (500) starts at (S501) and proceeds to (S510).

[0080] At (S510), the value of the LPS range (represented by the variable ivlLpsRange) is derived. In the example, given the current value of ivlCurrRange, the variable qRangeIdx is derived as equation (3): qRangeIdx = ivlCurrRange >> 5 Equation (3)

[0081] Then, given qRangeIdx, pStateIdx0, and pStateIdx1 (associated with ctxTable and ctxIdx), the value of the MPS (denoted by valMps) and the value of the LPS range (denoted by ivlLpsRange) are derived, for example, according to equations (4), (5), and (6): pState = pStateIdx1 + 16×pStateIdx0 Equation (4) valMps=pState>>14 Equation (5) ivlLpsRange=(qRangeIdx×((valMps?32767-pState:pState)>> 9 )>>1)+4 Equation (6)

[0082] Additionally, binary is assumed to be MPS, and the variable ivlCurrRange is set equal to (ivlCurrRange - ivlLpsRange).

[0083] At (S520), if the variable ivlOffset is greater than or equal to ivlCurrRange, the assumption (Binary is MPS) is not true, and the process proceeds to (S530); otherwise, the process proceeds to (S540).

[0084] At (S530), the binary is LPS, and therefore the binary value (represented by the variable binVal) is set equal to (1-valMps), the lower boundary of the current range and the current range are updated accordingly. In the example, the variable ivlOffset is decremented by ivlCurrRange, and ivlCurrRange is set equal to ivlLpsRange.

[0085] At (S540), the binary is MPS, and the variable binVal is set equal to valMps.

[0086] At (S550), a state transition is performed. Probabilistic state information, such as two probabilities pStateIdx0 and pStateIdx1, is updated based on the decoded value binVal.

[0087] In some examples, two adaptation variables shift0 and shift1 are derived from the shiftIdx value associated with ctxTable and ctxIdx, as shown in equations (7) and (8): shift0 = ( shiftIdx >> 2 ) + 2 Equation (7) shift1 = ( shiftIdx & 3 ) + 3 + shift0 Equation (8)

[0088] The probability state information is updated according to the decoded value binVal, for example, two probabilities pStateIdx0 and pStateIdx1, as shown by equations (9) and (10): pStateIdx0=pStateIdx0-(pStateIdx0>>shift0)+(1023× binVal>>shift0) Equation (9) pStateIdx1=pStateIdx1-(pStateIdx1>>shift1)+(16383× binVal>>shift1) Equation (10)

[0089] At (S560), for example, when the current range ivlCurrRange is less than 256, a renormalization step may be performed. Then, the process (500) proceeds to (S599) and terminates.

[0090] It should be noted that the probability state can be initialized at the beginning of each slice. In some examples (e.g., HEVC and VVC), the CABAC engine can have a QP-related initialization process called at the beginning of each slice. In an example, given the initial value of the luma QP of the slice, the initial probability state of the context model (denoted as preCtxState) is derived according to equations (11) to (15): slopeIdx = initValue >> 3 Equation (11) n = ( offsetIdx << 3 ) + 7 Equation (14) preCtxState=Clip3(1,127,((m×(Clip3(0,63,SliceQp Y )-16))>>1)+n) Equation (15) The variable initValue is a 6-bit variable obtained from the table associated with the variables ctxTable and ctxIdx, SliceQ pY Represents the slice QP value, slopeIdx and offsetIdx are limited to 3 bits. The probability state preCtxState directly represents the probability in the linear domain. It should be noted that Clip3(x,y,z) is a function that clips z in the range of [x,y].

[0091] It should be noted that in some examples, appropriate shift operations may be performed on preCtxState to generate input to the arithmetic decoding engine (for performing the arithmetic decoding step). In some examples, the mapping from the logarithmic domain to the linear domain and the table (400) are not required. In some examples, the two probabilities pStateIdx0 and pStateIdx1 may be derived from the initialization using shift operations, for example according to equations (16) and (17): pStateIdx0 = preCtxState << 3 Equation (16) pStateIdx1 = preCtxState << 7 Equation (17)

[0092] In some examples, for the decoding process of CABAC, the input of the decoding process can be all binary strings of syntax elements. The binary strings of syntax elements can be obtained based on a binarization process. The output of the decoding process can be the value of the syntax element. The decoding process can specify how to parse each binary of the binary string of each syntax element. After parsing each binary, the resulting binary string (or the parsed binary string) can be compared with all binary strings of syntax elements obtained based on the binarization process. The comparison result can be determined as follows: (1) If the parsed binary string (or the resulting binary string) is equal to one of the binary strings, the corresponding value of the syntax element is output. (2) Otherwise (for example, the parsed binary string is not equal to one of the binary strings), parse the next bit. When parsing each binary, the variable binIdx is set to equal to 0 and increases by 1. The parsing of each binary is specified by two ordered steps: (1) The derivation process of ctxTable, ctxIdx and bypassFlag can be called with binIdx as input and ctxTable, ctxIdx and bypassFlag as output. (2) The arithmetic decoding process can be called with ctxTable, ctxIdx and bypassFlag as input and binary value as output.

[0093] In the process of deriving ctxTable, ctxIdx and bypassFlag, the input of the deriving process can be the position of the current binary in the binary string. The position of the current binary in the binary string can be represented as a binary index (such as binIdx). The output of the deriving process can be ctxTable, ctxIdx and bypassFlag. The values ​​of ctxTable, ctxIdx and bypassFlag can be based on the table (such as Figure 6 It is obtained by the entry (or value) of the binIdx of the corresponding syntax element in Table (600) in .

[0094] Figure 6 A table (600) for assigning ctxInc to syntax elements with context coded bins is shown.

[0095] If the entry in table (600) is not equal to "bypass", "terminate", or "na", the value of binldx may be decoded by calling the DecodeDecision process. As shown in table (600), the variable ctxInc may be specified by the corresponding entry in table (600). When more than one value is listed for binldx in table (600), the allocation process of ctxInc for the corresponding binldx may be further specified by the derivation process of ctxInc. The variable ctxIdxOffset may be set equal to the current value of initType and the minimum value of ctxIdx of the current syntax element. ctxIdx may be set equal to the sum of ctxInc and ctxIdxOffset. bypassFlag may be set equal to 0.

[0096] If the entry in the table (600) is equal to "bypass", then the value of binldx may be decoded by calling the DecodeBypass procedure. Therefore, ctxTable may be set equal to 0. ctxIdx may be set equal to 0. bypassFlag may be set equal to 1.

[0097] If the entry in the table (600) is equal to "terminate", the value of binldx may be decoded by calling the DecodeTerminate procedure. Therefore, ctxTable may be set equal to 0 and ctxIdx may be set equal to 0. bypassFlag may be set equal to 0.

[0098] If an entry in table (600) is equal to "na", then the value of binldx does not appear for the corresponding syntax element.

[0099] In some examples, the derivation process of ctxInc can use the left syntax element and the upper syntax element. The input of the derivation process of ctxInc can be the luma position (x0, y0) of the upper left luma sample of the specified current luma block relative to the upper left sample of the current picture, the color component cIdx, the current decoding quadtree depth cqtDepth, the dual tree channel type chType, the width cbWidth and height cbHeight of the current decoding block in luma samples, and the variables derived in the decoding tree semantics. These variables may include allowSplitBtVer, allowSplitBtHor, allowSplitTtVer, allowSplitTtHor, and allowSplitQt. The output of the derivation process of ctxInc can be the context index ctxInc.

[0100] In some examples, the left position (xNbL, yNbL) may be set equal to (x0-1, y0), and the neighboring block availability derivation process may be called based on: the position (xCurr, yCurr) being set equal to (x0, y0), the neighboring position (xNbY, yNbY) being set equal to (xNbL, yNbL), checkPredModeY being set equal to FALSE, and cIdx being set as input. The output of the neighboring block availability derivation process may be assigned to the variable availableL.

[0101] In some embodiments, the above position (xNbA, yNbA) can be set equal to (x0, y0-1), and the neighboring block availability derivation process can be called based on the following: the position (xCurr, yCurr) is set equal to (x0, y0), the neighboring position (xNbY, yNbY) is set equal to (xNbA, yNbA), checkPredModeY is set equal to FALSE, and cIdx is set as input. The output of the neighboring block availability derivation process can be assigned to the variable availableA.

[0102] In an example, the position (xCtb, yCtb) (e.g., the CTB of the current block) may be set equal to (x0>>CtbLog2SizeY, y0>>CtbLog2SizeY), the position (xCtbA, yCtbA) (e.g., the CTB of the upper neighboring block) may be set equal to (xNbA>>CtbLog2SizeY, yNbA>>CtbLog2SizeY), and the position (xCtbL, yCtbL) (e.g., the CTB of the left neighboring block) may be set equal to (xNbL>>CtbLog2SizeY, yNbL>>CtbLog2SizeY). Then ctxInc may be determined based on the upper neighboring block and the left neighboring block.

[0103] Figure 7 A table (700) is shown for illustrating the assignment of ctxInc using the left syntax element and the above syntax element. The assignment of ctxInc may be specified by condL and condA specified in the table (700) as follows.

[0104] In some examples, for the syntax elements alf_ctb_flag[cIdx][xCtb][yCtb], alf_ctb_cc_cb_idc[xCtb][yCtb], alf_ctb_cc_cr_idc[xCtb][yCtb], split_qt_flag, split_cu_flag, cu_skip_flag[x0][y0], pred_mode_ibc_flag[x0][y0], intra_mip_flag, inter_affine_flag[x0][y0], and merge_subblock_flag[x0][y0], ctxInc may be determined according to equation (18) as follows: ctxInc=(condL&&availableL)+(condA&&availableA)+ctxSetIdx×3 Equation (18)

[0105] In some examples, for the syntax elements pred_mode_flag[x0][y0] and non_inter_flag, ctxInc may be determined according to equation (19) as follows: ctxInc = (condL && availableL) | | (condA && availableA) Equation (19)

[0106] In a related example, it may be inefficient to use information only from the upper and left neighboring blocks of the current block to derive the CABAC context model of the current block (eg, for inter_affine_flag or merge_subblock_flag).

[0107] According to an aspect of the present disclosure, the context model of some syntax elements of the current block may depend on information of more than two spatial neighboring blocks of the current block and / or temporal neighboring blocks (also referred to as co-located blocks) of the current block. The syntax elements may include an inter-frame affine flag (e.g., inter_affine_flag) and / or a sub-block merge flag (e.g., merge_subblock_flag) and / or a LIC flag (e.g., lic_flag). It should be noted that the syntax elements may include other suitable syntax elements in Table (700).

[0108] For simplicity and clarity, some embodiments of the present disclosure may be provided based on specific syntax elements, such as inter_affine_flag as an exemplary syntax element.

[0109] In some examples, one or more additional context increments (e.g., ctxInc) may be used to determine the syntax elements of the current block. ctxInc may indicate a context index (e.g., ctxIdx) of a context model. For example, ctxIdx may be set equal to the sum of ctxInc and ctxIdxOffset. ctxIdxOffset may be a fixed value that depends on the type of syntax element and the type of slice (I, B, and P). The context increment (e.g., ctxInc) may be calculated using information such as the size of the current block, the type of the current block, and information about neighboring blocks of the current block. Thus, the context model of the syntax element may be derived depending on the information of the M spatial neighboring blocks of the current block, where M may be a positive integer.

[0110] In an embodiment, M may be equal to 5. The 5 spatial neighbors of the current block may be as follows: Figure 8 As shown. Figure 8 As shown, the current block (802) may include an upper right (B0) neighboring block, an upper (B1) neighboring block, an upper left (B2) neighboring block, a left (A1) neighboring block, and a lower left (A0) neighboring block. Each neighboring block may have various sizes. For example, Figure 8 Each neighboring block in may include 4×4 luma samples.

[0111] In an embodiment, M may be equal to the number of all sub-blocks from the lower left (LB) position to the upper left (AL) position and from the upper left (AL) position to the upper right (AR) position, which may be as follows: Fig. 9 As shown. Fig. 9 As shown, the current block (902) may include a column of neighboring blocks (e.g., LB, L0, L1, L2, L3, AL) adjacent to the left side of the current block (902), and a row of neighboring blocks (AL, A3, A2, A1, A0, and AR) adjacent to the top side of the current block (902). Each neighboring block may have various sizes. For example, Fig. 9 Each neighboring block in may include 4×4 luma samples.

[0112] In an embodiment, the additional ctxInc of inter_affine_flag may be as shown in Table A. As shown in Table A, four values ​​of ctxInc (eg, 0, 1, 2, and 3) may be provided to the first context decoding bin (eg, binIdx=0) of inter_affine_flag. Table A. Allocation of ctxInc to syntax elements with context decoding binary

[0113] In another embodiment, the additional ctxInc of merge_subblock_flag may be as shown in Table B. As shown in Table B, four values ​​of ctxInc (eg, 0, 1, 2, and 3) may be provided to the first context decoding bin (eg, binIdx=0) of merge_subblock_flag. Table B. Allocation of ctxInc to Syntax Elements with Context Coding Binary

[0114] In an embodiment, the ctxInc of inter_affine_flag may be set to 3 based on a combination of the following two conditions: (1) 5 (e.g. Figure 8 N or more of the available spatial neighbors (A0, A1, B0, B1, and B2) are available (e.g., as determined by the neighbor block availability derivation process, as in subclause 6.4.4 of the VVC specification document). In the example, N is equal to 4. (2) For each available spatial neighbor, a condition based on a merge subblock flag (e.g., MergeSubblockFlag) and an inter-frame affine flag (InterAffineFlag) is false, as shown in equation (20): (MergeSubblockFlag[xNb][yNb]||InterAffineFlag[xNb][yNb]) Equation (20) Otherwise, if at least one of the two conditions is not met, the value of ctxInc of inter_affine_flag can be derived based on the conditions of the left neighbor and the upper neighbor (respectively, condL and condA). condL and condA can be determined based on table (700). ctxInc can be determined to be 0, 1, or 2, for example, based on equation (18) (where the value of ctxSetIdx is 0).

[0115] For example, as shown in Table (700), for inter_affine_flag, condL corresponds to MergeSubblockFlag[xNbL][yNbL]||InterAffineFlag[xNbL][yNbL], and condA corresponds to MergeSubblockFlag[xNbA][yNbA]||InterAffineFlag[xNbA][yNbA]. In an example, when (condL&&availableL) is satisfied and (condA&&avaialbeA) is not satisfied, ctxInc may be determined to be 0. In an example, when (condL&&availableL) is not satisfied and (condA&&avaialbeA) is satisfied, ctxInc may be determined to be 1. In an example, when both (condL&&availableL) and (condA&&avaialbeA) are satisfied, ctxInc may be determined to be 2.

[0116] In an embodiment, the ctxInc of merge_subblock_flag may be set to 3 based on a combination of the following two conditions: (1) 5 (e.g. Figure 8 N or more of the available spatial neighbors (A0, A1, B0, B1, and B2) are available (e.g., determined by the derivation process of neighbor block availability, as in subclause 6.4.4 of the VVC specification document). In the example, N is equal to 4. (2) For each available spatial neighbor, the condition in equation (20) is false.

[0117] Otherwise, if at least one of the two conditions is not met, the value of ctxInc of merge_subblock_flag can be derived based on the conditions of the left neighbor and the upper neighbor (respectively, condL and condA). condL and condA can be determined based on table (700). ctxInc can be determined to be 0, 1, or 2 based on, for example, equation (18) (where the value of ctxSetIdx is 0).

[0118] In an embodiment, the context model of the LIC flag (e.g., lic_flag) of the current block may be derived based on more than two neighboring blocks of the current block. If the value of lic_flag is true, the LIC flag may indicate that LIC is applied to the current block. Otherwise, if the value of lic_flag is false, LIC is not applied to the current block.

[0119] Table C shows that four ctxInc values ​​may be provided to the first context decoding bin (eg, binIdx=0) of lic_flag. Table D shows the conditions of the left neighboring block and the conditions of the upper neighboring block of the current block. Table C. Allocation of ctxInc to Syntax Elements with Context Coding Binary Table D. Specification of ctxInc using left and above syntax elements Syntax elements cond condA ctxSetIdx lic_flag[x0][y0] LicFlag[xNbL][yNbL] LicFlag[xNbA][yNbA] 0 In this example, the ctxInc of lic_flag can be set to 3 based on a combination of the following two conditions: (1) 5 (e.g. Figure 8 N or more of the spatial neighbors (A0, A1, B0, B1, and B2) are available. In the example, N is equal to 4. (2) The LIC flag of each available spatial neighbor is false. Otherwise, if at least one of the two conditions is not met, the ctxInc of lic_flag can be derived based on the conditions of the left neighbor and the upper neighbor (respectively, condL and condA). condL and condA can be determined based on Table D. ctxInc can be determined to be 0, 1, or 2, for example, according to equation (18) (where the value of ctxSetIdx is 0).

[0120] In an embodiment, a similar context modeling method may be applied to other syntax elements that currently have ctxInc derived using the left syntax element and the above syntax element, such as non_inter_flag, cu_skip_flag, pred_mode_flag, intra_mip_flag, etc. Therefore, the context models of non_inter_flag, cu_skip_flag, pred_mode_flag, and intra_mip_flag may be derived based on more than two neighboring blocks.

[0121] According to some aspects of the present disclosure, a context model of a syntax element may be derived from one or more temporally co-located blocks of a current block. In some implementations, a context model of a syntax element may be derived based on information of a spatially neighboring block of the current block and N temporally co-located blocks. Example values ​​of N include, but are not limited to, 1, 2, 3, 4, ...

[0122] In an embodiment, the N temporally co-located blocks refer to N sub-blocks located at N predefined relative positions of the co-located block having the same coordinates of the current block in the reference picture.

[0123] Fig.10A diagram of a current picture (1010) and a reference picture (1020) in some examples is shown. The current picture (1010) includes a current block (1011) being decoded. In the reference picture, a co-located block (1021) can be positioned to have the same coordinates as the current block (1011). Based on the co-located block (1021), N sub-blocks having predefined relative positions of the co-located block (1021) can be determined. For example, N is equal to 2, and the two predefined relative positions include the middle position shown by sub-block C0 and the lower right position shown by sub-block C1. Then, the N sub-blocks (also referred to as temporal neighboring blocks, temporal co-located blocks), such as sub-block C0 and sub-block C1, can be used to derive a context model. It should be noted that the temporal co-located blocks can be used together with or without spatial neighboring blocks to derive a context model.

[0124] In some examples, M spatial neighboring blocks and N temporal neighboring blocks are scanned, and the context model is derived by the number of blocks associated with a particular value of the syntax element. For example, when a flag (e.g., pred_mode_flag) indicating whether the current block is only intra-coded is decoded, the context model is derived based on the number of spatial and temporal neighboring blocks that are intra-coded.

[0125] Fig.11 A flow chart outlining a process (1100) according to an embodiment of the present disclosure is shown. The process (1100) may be used in a video decoder. In various embodiments, the process (1100) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video decoder (110), a processing circuit system that performs the functions of a video decoder (210), etc. In some embodiments, the process (1100) is implemented as software instructions, so when the processing circuit system executes the software instructions, the processing circuit system performs the process (1100). The process starts at (S1101) and proceeds to (S1110).

[0126] At (S1110), decoded information of a current block in a current picture may be received from a decoded video bitstream. The decoded information includes syntax elements associated with the current block, the syntax elements indicating decoding parameters of the current block.

[0127] At (S1120), a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element is determined based on at least a temporally co-located block of the current block.

[0128] At (S1130), the syntax elements are decoded based on the CABAC context model.

[0129] At (S1140), the current block is reconstructed based on the syntax elements decoded based on the CABAC context model.

[0130] In some examples, the temporally co-located block is located in a reference picture of the current picture at the same coordinates as the current block.

[0131] In some examples, the CABAC context model is determined based on one or more sub-blocks located at predefined positions relative to the temporally co-located block. In an example, the CABAC context model is determined based on at least a first sub-block at a middle position relative to the temporally co-located block and a second sub-block at a lower right position relative to the temporally co-located block.

[0132] In some examples, a CABAC context model associated with a syntax element is determined based on at least spatial neighboring blocks and at least temporal co-located blocks of the current block. For example, at least spatial neighboring blocks and at least temporal co-located blocks are scanned to determine the number of blocks that meet the requirements in at least spatial neighboring blocks and at least temporal co-located blocks. Then, the CABAC context model is determined based on the number of blocks that meet the requirements. In an example, the syntax element is a flag indicating whether the current block is only intra-coded. Then, at least spatial neighboring blocks and at least temporal co-located blocks are used to determine the number of blocks that are intra-coded in at least spatial neighboring blocks and at least temporal co-located blocks, and the CABAC context model used to derive the flag is determined based on the number of blocks.

[0133] It should be noted that the syntax element can be one of the following: inter-frame affine flag, merge sub-block flag, local illumination compensation flag, non-inter-frame flag, decoding unit skip flag, prediction mode flag or inter-frame matrix-based intra-prediction (mip) flag.

[0134] Then, the process proceeds to (S1199) and terminates.

[0135] The process (1100) may be adjusted appropriately. Steps in the process (1200) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0136] Fig.12 A flow chart outlining a process (1200) according to an embodiment of the present disclosure is shown. The process (1200) may be used in a video encoder. In various embodiments, the process (1200) is performed by a processing circuit system, such as a processing circuit system that performs the functions of a video encoder (103), a processing circuit system that performs the functions of a video encoder (303), etc. In some embodiments, the process (1200) is implemented in software instructions, so that when the processing circuit system executes the software instructions, the processing circuit system performs the process (1200). The process starts at (S1201) and proceeds to (S1210).

[0137] At (S1210), for a current block in a current picture, a CABAC context model for coding a syntax element associated with the current block is determined from a plurality of candidate CABAC context models based on at least a temporally co-located block of the current block.

[0138] At (S1220), syntax elements associated with the current block are encoded using context coding based on the determined CABAC context model.

[0139] At (S1230), coded information of the current block is generated based on the determined CABAC context model, the coded information including context-coded syntax elements.

[0140] In some examples, the temporally co-located block is located in a reference picture of the current picture at the same coordinates as the current block. In some embodiments, the CABAC context model is determined based on one or more sub-blocks located at predefined positions relative to the temporally co-located block. In an example, the CABAC context model is determined based on at least a first sub-block at a middle position relative to the temporally co-located block and a second sub-block at a lower right position relative to the temporally co-located block.

[0141] In some examples, a CABAC context model associated with a syntax element is determined based on at least spatial neighboring blocks and at least temporal co-located blocks of the current block. In some embodiments, at least spatial neighboring blocks and at least temporal co-located blocks are scanned to determine the number of blocks that meet the requirements in at least spatial neighboring blocks and at least temporal co-located blocks, and the CABAC context model is determined based on the number of blocks that meet the requirements. In an example, the syntax element is a flag indicating whether the current block is only intra-coded, and then at least spatial neighboring blocks and at least temporal co-located blocks are scanned to determine the number of blocks that are intra-coded in at least spatial neighboring blocks and at least temporal co-located blocks, and the CABAC context model is determined based on the number of blocks.

[0142] It should be noted that the syntax element can be one of the following: inter-frame affine flag, merge sub-block flag, local illumination compensation flag, non-inter-frame flag, decoding unit skip flag, prediction mode flag or inter-frame matrix-based intra-prediction (mip) flag.

[0143] Then, the process proceeds to (S1299) and terminates.

[0144] The process (1200) may be adjusted appropriately. Steps in the process (1200) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0145] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.13 A computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0146] Computer software may be encoded using any suitable machine code or computer language, which may be assembled, compiled, linked, or the like to create code comprising instructions, which may be executed directly by one or more computer central processing units (CPU), graphics processing units (GPU), etc., or by interpretation, microcode execution, or the like.

[0147] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0148] Fig.13 The components for the computer system (1300) shown in the example are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one component or combination of components shown in the exemplary embodiment of the computer system (1300).

[0149] The computer system (1300) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not depicted). Human-machine interface devices may also be used to capture specific media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0150] The input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (1301), mouse (1302), touchpad (1303), touch screen (1310), data gloves (not shown), joystick (1305), microphone (1306), scanner (1307), camera (1308).

[0151] The computer system (1300) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more senses of a human user through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include: tactile output devices (e.g., tactile feedback through a touch screen (1310), a data glove (not shown), or a joystick (1305), but there may also be tactile feedback devices that are not used as input devices); audio output devices (e.g., speakers (1309), headphones (not depicted)); visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may be capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted); holographic displays and smoke generators (not depicted)); and printers (not depicted).

[0152] The computer system (1300) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1320) with CD / DVD etc. media (1321), thumb drives (1322), removable hard drives or solid-state drives (1323), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD based devices such as security dongles (not depicted), etc.

[0153] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0154] The computer system (1300) may also include an interface (1354) to one or more communication networks (1355). The network may be, for example, a wireless network, a wired network, an optical network. The network may also be local, wide, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include: local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; in-vehicle and industrial networks including CANBus, etc. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1349) (e.g., a USB port of the computer system (1300)); other networks are typically integrated into the core of the computer system (1300) by attaching to the system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). The computer system (1300) can use any of these networks to communicate with other entities. Such communications may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way, such as to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used on each of these networks and network interfaces as described above.

[0155] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be attached to the core ( 1340 ) of the computer system ( 1300 ).

[0156] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1343), hardware accelerators for certain tasks (1344), graphics adapters (1350), etc. These devices, along with read-only memory (ROM) (1345), random access memory (1346), internal mass storage devices such as internal non-user accessible hard drives, SSDs, etc. (1347), may be connected via a system bus (1348). In some computer systems, the system bus (1348) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or to the core's system bus (1348) via a peripheral bus (1349). In an example, the screen (1310) may be connected to a graphics adapter (1350). The architecture of the peripheral bus includes PCI, USB, and the like.

[0157] The CPU (1341), GPU (1342), FPGA (1343) and accelerator (1344) can execute certain instructions, which can be combined to form the aforementioned computer code. The computer code can be stored in ROM (1345) or RAM (1346). Transient data can also be stored in RAM (1346), while permanent data can be stored in, for example, an internal mass storage device (1347). Fast storage and retrieval of any of the memory devices can be achieved by using a cache memory, which can be closely associated with one or more CPUs (1341), GPUs (1342), mass storage devices (1347), ROMs (1345), RAMs (1346), etc.

[0158] The computer readable medium may have thereon computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or the medium and computer code may be of a type well known and available to those skilled in the art of computer software.

[0159] As an example and not a limitation, a computer system (1300) having an architecture, particularly a core (1340), can provide functionality due to a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such a computer-readable medium can be a medium associated with a user-accessible mass storage device as described above, and certain storage devices of the core (1340) having a non-transient nature, such as a mass storage device (1347) or ROM (1345) inside the core. Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (1340). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core (1340), particularly the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described herein, including defining a data structure stored in the RAM (1346) and modifying such a data structure according to a process defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1344)) that may operate in place of or in conjunction with software to perform a particular process or particular portion of a particular process described herein. Where appropriate, reference to software may encompass logic, and vice versa. Where appropriate, reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0160] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various substitution equivalents that fall within the scope of the present disclosure. It will therefore be appreciated that those skilled in the art will be able to devise a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for video decoding, include: Receiving decoded information of a current block in a current picture from a decoded video bitstream, the decoded information comprising a syntax element associated with the current block, the syntax element indicating a decoding parameter of the current block; determining a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element based on at least a temporally co-located block of the current block, the temporally co-located block being in a different picture than the current picture; Decoding the syntax element based on the CABAC context model; as well as The current block is reconstructed based on the syntax elements decoded based on the CABAC context model.

2. The method according to claim 1, in, The temporally co-located block is located in a reference picture of the current picture at the same coordinates as the current block.

3. The method according to claim 2, in, Determining the CABAC context model further includes: The CABAC context model is determined based on one or more sub-blocks located at predefined positions relative to the temporally co-located block.

4. The method according to claim 3, in, Determining the CABAC context model further includes: The CABAC context model is determined based on at least a first subblock at a middle position relative to the temporally co-located block and a second subblock at a lower right position relative to the temporally co-located block.

5. The method according to claim 1, in, Determining the CABAC context model further includes: The CABAC context model associated with the syntax element is determined based on at least spatial neighboring blocks and at least the temporal co-located block of the current block.

6. The method according to claim 5, in, Determining the CABAC context model further includes: Scanning at least the spatially neighboring blocks and at least the temporally co-located blocks to determine the number of blocks that meet the requirement among at least the spatially neighboring blocks and at least the temporally co-located blocks; and The CABAC context model is determined based on the number of the blocks that meet the requirement.

7. The method according to claim 6, in, The syntax element is a flag indicating whether the current block is only intra-coded, and the method comprises: Scanning at least the spatially neighboring blocks and at least the temporally co-located blocks to determine the number of intra-coded blocks in at least the spatially neighboring blocks and at least the temporally co-located blocks; and The CABAC context model used to derive the flag is determined based on the number of blocks.

8. The method according to claim 1, in, The syntax element is one of: an inter affine flag, a merge subblock flag, a local illumination compensation flag, a non-inter flag, a coding unit skip flag, a prediction mode flag, or an inter matrix-based intra prediction (mip) flag.

9. An apparatus comprising processing circuitry, the processing circuitry being configured to: Receiving decoded information of a current block in a current picture from a decoded video bitstream, the decoded information comprising a syntax element associated with the current block, the syntax element indicating a decoding parameter of the current block; determining a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element based on at least a temporally co-located block of the current block, the temporally co-located block being in a different picture than the current picture; Decoding the syntax element based on the CABAC context model; as well as The current block is reconstructed based on the syntax elements decoded based on the CABAC context model.

10. The device according to claim 9, in, The temporally co-located block is located in a reference picture of the current picture at the same coordinates as the current block.

11. The device according to claim 10, in, The processing circuit system is configured to: The CABAC context model is determined based on one or more sub-blocks located at predefined positions relative to the temporally co-located block.

12. The device according to claim 11, in, The processing circuit system is configured to: The CABAC context model is determined based on at least a first subblock at a middle position relative to the temporally co-located block and a second subblock at a lower right position relative to the temporally co-located block.

13. The device according to claim 9, in, The processing circuit system is configured to: The CABAC context model associated with the syntax element is determined based on at least spatial neighboring blocks and at least the temporal co-located block of the current block.

14. The device according to claim 13, in, The processing circuit system is configured to: Scanning at least the spatially adjacent blocks and at least the temporally co-located blocks to determine the number of blocks that meet the requirement in at least the spatially adjacent blocks and at least the temporally co-located blocks; as well as The CABAC context model is determined based on the number of the blocks that meet the requirement.

15. The device according to claim 14, in, The syntax element is a flag indicating whether the current block is intra-coded only, and the processing circuitry is configured to: Scanning at least the spatially neighboring blocks and at least the temporally co-located blocks to determine the number of blocks to be intra-coded in at least the spatially neighboring blocks and at least the temporally co-located blocks; as well as The CABAC context model used to derive the flag is determined based on the number of blocks.

16. The device according to claim 9, in, The syntax element is one of: an inter affine flag, a merge subblock flag, a local illumination compensation flag, a non-inter flag, a coding unit skip flag, a prediction mode flag, or an inter matrix-based intra prediction (mip) flag.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: Receiving decoded information of a current block in a current picture from a decoded video bitstream, the decoded information comprising a syntax element associated with the current block, the syntax element indicating a decoding parameter of the current block; determining a context-based adaptive binary arithmetic coding (CABAC) context model associated with the syntax element based on at least a temporally co-located block of the current block, the temporally co-located block being in a different picture than the current picture; Decoding the syntax element based on the CABAC context model; as well as The current block is reconstructed based on the syntax elements decoded based on the CABAC context model.

18. The non-transitory computer readable storage medium of claim 17, in, The temporally co-located block is located in a reference picture of the current picture at the same coordinates as the current block.

19. The non-transitory computer readable storage medium of claim 18, in, The instructions cause the at least one processor to: The CABAC context model is determined based on one or more sub-blocks located at predefined positions relative to the temporally co-located block.

20. The non-transitory computer readable storage medium of claim 19, in, The instructions cause the at least one processor to: The CABAC context model is determined based on at least a first subblock at a middle position relative to the temporally co-located block and a second subblock at a lower right position relative to the temporally co-located block.