Method, device and storage medium for decoding a current block in a video

Abstract

The present disclosure relates to a method, device and storage medium for decoding a current block in a video. For example, a method for decoding a current block in a video is disclosed. The method may include: determining a zero skip flag of at least one neighboring block of the current block as a reference zero skip flag; determining a prediction mode for the current block as an intra-frame prediction mode or an inter-frame prediction mode; deriving at least one context for decoding the zero skip flag of the current block based on the reference zero skip flag and the prediction mode of the current block; and decoding the zero skip flag of the current block according to the at least one context.

Description

[0001] Incorporation by Reference

[0002] This application is based on and claims priority to U.S. non-provisional patent application No. 17 / 573,299 filed on January 11, 2022, and U.S. provisional application No. 63 / 209,261, entitled “Zero Residual Flag Encoding and Decoding” filed on June 10, 2021. The entire contents of these two applications are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to a set of advanced video encoding / decoding techniques, and more particularly, to an improved encoding and decoding scheme for a zero residue flag or a zero coefficient flag. Background Art

[0004] The background description provided herein is intended to present the background of the present application as a whole. The extent to which the work of the presently named inventors described in the background section and various aspects of this specification is performed does not indicate that it is prior art at the time of filing this application, and it is never explicitly or implicitly admitted that it is prior art for this application.

[0005] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. An uncompressed digital video may include a series of pictures, each picture having a spatial dimension of, for example, 1920×1080 luminance samples and associated fully sampled or subsampled chrominance samples. The series of pictures has a fixed or variable picture rate (or frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements. For example, a video with a pixel resolution of 1920×1080, a frame rate of 60 frames per second, and a chrominance subsampling of 4:2:0, with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0006] One of the goals of video encoding and decoding is to reduce the redundant information of the uncompressed input video signal through compression. Video compression can help reduce the requirements for the above-mentioned bandwidth and / or storage space, in some cases by two or more orders of magnitude. Lossless compression and lossy compression, as well as a combination of the two, can be used. Lossless compression refers to the technology of reconstructing an exact copy of the original signal from the compressed original signal through the decoding process. Lossy compression refers to the encoding / decoding process in which the original video information is not completely preserved during encoding and cannot be completely restored during decoding. When lossy compression is used, the reconstructed signal may be different from the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal usable for the intended application, despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of tolerable distortion depends on the application. For example, users of some consumer video streaming applications can tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a specific encoding algorithm can be selected or adjusted to reflect various distortion tolerances: higher tolerable distortion generally allows encoding algorithms that produce higher losses and higher compression ratios.

[0007] Video encoders and decoders may utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transforms, quantization, and entropy coding.

[0008] Video codec techniques may include known intra-frame coding techniques. In intra-frame coding, sample values ​​are represented without reference to samples or other data of previously reconstructed reference pictures. In some video codecs, pictures are spatially subdivided into sample blocks. When all sample blocks are encoded in intra-frame mode, the picture may be referred to as an intra-frame picture. Intra-frame pictures and their derivatives (e.g., independent decoder refresh pictures) can be used to reset the decoder state, and can therefore be used as the first picture in a coded video bitstream and a video session, or as a still image. The samples of the intra-frame predicted block can then be transformed into the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-frame prediction represents a technique for minimizing sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform, and the smaller the AC coefficient, the fewer bits are needed to represent the block after entropy coding at a given quantization step size.

[0009] As known from coding techniques such as MPEG-2, conventional intra-frame coding does not use intra-frame prediction. However, some newer video compression techniques include attempts to encode / decode blocks based on, for example, surrounding sample data and / or metadata that are obtained during spatially adjacent encoding and / or decoding and precede the data block being encoded or decoded within the frame in decoding order. Such techniques are hereinafter referred to as "intra-frame prediction" techniques. Note that, in at least some cases, intra-frame prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0010] There can be many different forms of intra-frame prediction. When more than one such technique is available in a given video coding technique, the technique used may be referred to as an intra-frame prediction mode. One or more intra-frame prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra-frame coding parameters for a video block may be contained in a mode codeword, which may be encoded separately or together. For a given mode, sub-mode, and / or parameter combination, which codeword is used can have an impact on the coding efficiency gain through intra-frame prediction, and the same is true for the entropy coding technique used to convert the codeword into a bitstream.

[0011] Some mode of intra prediction was introduced with H.264, modified in H.265, and further modified in newer coding techniques such as Joint Detection Mode (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Typically, for intra prediction, the predictor block can be formed using the values ​​of neighboring samples that have become available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or row can be copied into the predictor block. A reference to the direction of use can be encoded in the bitstream, or it can be predicted itself.

[0012] refer to Figure 1A , depicted at the bottom right is a subset of the 9 predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the directions according to which the sample at 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample (101) is predicted based on one or more neighboring samples to the upper right at an angle of 45 degrees to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted based on one or more neighboring samples to the lower left of sample (101) at an angle of 22.5 degrees to the horizontal.

[0013] Still reference Figure 1A, a square block (104) comprising 4×4 samples is shown in the upper left (indicated by the thick dashed line). The square block (104) consists of 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (starting from the top) and the first sample in the X dimension (starting from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y dimension and the X dimension. Since the block is a 4×4 size sample, S44 is located in the lower right corner. Example reference samples following a similar numbering scheme are also shown. The reference samples are labeled with "R" and their Y position (e.g., row index) and X position (e.g., column index) relative to the block (104). In H.264 and H.265, neighboring prediction samples adjacent to the block being reconstructed are used.

[0014] The intra-picture prediction of block 104 can start by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the encoded video code stream includes signaling indicating the prediction direction of arrow (102) for this block 104 - that is, predicting samples based on one or more prediction samples to the upper right at a 45 degree angle to the horizontal direction. In such a case, samples S41, S32, S23, and S14 are predicted based on the same reference sample R05. Sample S44 is then predicted based on reference sample R08.

[0015] In some cases, such as by interpolation, the values ​​of multiple reference samples may be combined in order to compute the reference sample, particularly when the direction is not divisible by 45 degrees.

[0016] As video coding technology continues to develop, the number of possible directions increases. For example, in H.264 (2003), 9 different directions can be used for intra-frame prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been conducted to help identify the most suitable intra-frame prediction directions, and certain techniques in entropy coding can be used to encode those most suitable directions with a small number of bits, accepting some bit costs for the directions. In addition, the direction itself can sometimes be predicted based on the neighboring directions used for intra-frame prediction of neighboring blocks that have already been decoded.

[0017] Figure 1B A schematic diagram (180) depicting 65 intra prediction directions according to JEM is shown to illustrate the increase in the number of prediction directions in various encoding techniques over time.

[0018] The manner in which the bits representing the intra-frame prediction direction are mapped to the prediction direction in the encoded video bitstream may vary with different video coding techniques; and may be, for example, a simple direct mapping from prediction direction to intra-frame prediction mode, to codewords, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there may be certain directions for intra-frame prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, those less probable directions will be represented by a larger number of bits than the more probable directions.

[0019] Inter-picture prediction or inter-frame prediction can be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) can be used for prediction of a newly reconstructed picture or picture part (e.g., block) after spatial shifting in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture in the current reconstruction. MV may have two dimensions, X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (approximately the time dimension).

[0020] In some video compression techniques, the current MV applicable to a certain area of ​​sample data can be predicted from other MVs, such as those other MVs related to other areas of sample data of the area in spatially adjacent reconstruction and preceding the current MV in decoding order. Doing so can significantly reduce the total amount of data required to encode the MV by relying on removing redundancy in the relevant MVs, thereby increasing compression efficiency. MV prediction can be performed efficiently, for example, because when encoding an input video signal derived from a camera (called natural video), there is a statistical possibility that a larger area than the area applicable to a single MV moves in a similar direction in the video sequence. Therefore, in some cases, similar motion vectors derived from MVs of adjacent areas can be used for prediction. This results in the actual MV of a given area being similar or identical to the MV predicted from the surrounding MVs. After entropy coding, such MVs can be represented with fewer bits than if the MV is directly encoded instead of predicted from one or more adjacent MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example due to rounding errors when calculating predicted values ​​from several surrounding MVs.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the multiple MV prediction mechanisms specified by H.265, this application describes a technique referred to below as "spatial merging".

[0022] Please refer to Figure 2 , the current block (201) includes samples that have been discovered by the encoder during the motion search process, and the samples can be predicted based on the previous block of the same size that has generated a spatial offset. In addition, the MV can be derived from metadata associated with one or more reference pictures instead of encoding the MV directly. For example, the MV is derived from the metadata of the nearest reference picture (in decoding order) using the MV associated with any of the five surrounding samples A0, A1 and B0, B1, B2 (corresponding to 202 to 206 respectively). In H.265, MV prediction can use the prediction value of the same reference picture also used by the neighboring blocks. Summary of the invention

[0023] Various aspects of the present disclosure provide methods, devices, and storage media for video encoding and decoding, for improving encoding and decoding schemes of zero residual or zero coefficient flags.

[0024] In some example embodiments, a method for decoding a current block in a video is disclosed. The method may include determining a zero skip flag of at least one neighboring block of the current block as a reference zero skip flag; determining a prediction mode for the current block as an intra prediction mode or an inter prediction mode; deriving at least one context for decoding the zero skip flag of the current block based on the reference zero skip flag and the prediction mode of the current block; and decoding the zero skip flag of the current block according to the at least one context.

[0025] In the above-mentioned embodiment, the at least one neighboring block includes a first neighboring block and a second neighboring block of the current block; and the reference zero skip flag includes a first reference zero skip flag and a second reference zero skip flag corresponding to the first neighboring block and the second neighboring block, respectively. In some embodiments, deriving the at least one context based on the reference zero skip flag and the prediction mode of the current block includes: in response to the prediction mode of the current block being an inter-frame prediction mode: deriving a first mode-related reference zero skip flag based on the first prediction mode of the first neighboring block and the first reference zero skip flag; deriving a second mode-related reference zero skip flag based on the second prediction mode of the second neighboring block and the second reference zero skip flag; and deriving the at least one context based on the first mode-related reference zero skip flag and the second mode-related reference zero skip flag. In some embodiments, deriving the first mode-related reference zero skip flag includes: in response to the first prediction mode being an inter-frame prediction mode, setting the first mode-related reference zero skip flag to the value of the first reference zero skip flag; and in response to the first prediction mode not being an inter-frame prediction mode, setting the first mode-related reference zero skip flag to a value indicating non-skipping. Deriving the second mode-related reference zero skip flag includes: in response to the second prediction mode being an inter-frame prediction mode, setting the second mode-related reference zero skip flag to the value of the second reference zero skip flag; and in response to the second prediction mode not being an inter-frame prediction mode, setting the second mode-related reference zero skip flag to a value indicating non-skipping.

[0026] In some of the above-mentioned embodiments, based on the first mode-related reference zero skip flag and the second mode-related reference zero skip flag, deriving the at least one context includes: deriving one context of the at least one context as the sum of the first mode-related reference zero skip flag and the second mode-related reference zero skip flag.

[0027] In some of the above implementations, the first neighboring block and the second neighboring block may include a top neighboring block and a left neighboring block of the current block, respectively.

[0028] In some of the above-mentioned embodiments, when the prediction mode of the first neighboring block is the intra-frame copy mode, the first prediction mode is regarded as the inter-frame prediction mode; and when the prediction mode of the second neighboring block is the intra-frame copy mode, the second prediction mode is regarded as the inter-frame prediction mode.

[0029] In some of the above embodiments, before signaling the zero skip flag of the current block, a prediction mode flag indicating whether the current block is an inter-coded block is signaled.

[0030] In some of the above embodiments, when the prediction mode associated with the current block is an intra block copy mode, the prediction mode of the current block is regarded as an inter prediction mode.

[0031] In some of the above embodiments, the at least one context includes a first context and a second context, and when the current block is intra-coded and inter-coded respectively, the first context and the second context are used to encode the zero skip flag of the current block.

[0032] In some other example embodiments, a method for encoding and decoding a current block in a video is disclosed. The method may include determining a zero skip flag of the current block to indicate whether the current block is an all-zero block; deriving a set of at least one context for decoding a prediction mode flag based on the value of the zero skip flag, the prediction mode flag indicating whether the current block is intra-frame decoded or inter-frame decoded; and decoding the prediction mode flag of the current block according to the set of at least one context.

[0033] In some of the above-mentioned embodiments, when the zero skip flag of the current block indicates that the current block is not an all-zero block, the set of at least one context includes a set of first contexts; and when the zero skip flag of the current block indicates that the current block is an all-zero block, the set of at least one context includes a set of second contexts different from the set of first contexts.

[0034] In some of the above embodiments, the set of the at least one context further depends on the value of at least one reference zero skip flag of at least one neighboring block of the current block.

[0035] In some other example embodiments, a method for decoding a current block in a video is disclosed. The method may include parsing a video bitstream to identify a zero skip flag of the current block; determining whether the current block is an all-zero block according to the zero skip flag; and in response to the current block being an all-zero block, setting a prediction mode flag of the current block to indicate an inter-frame prediction mode, rather than parsing the prediction mode flag from the video bitstream.

[0036] In some of the above embodiments, the method further includes using the prediction mode flag of the current block in decoding a neighboring block of the current block.

[0037] Aspects of the present disclosure also provide a device or apparatus having a memory for storing computer instructions and a processor configured to execute the computer instructions to implement any of the above methods.

[0038] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions, which, when executed by a computer for video decoding and / or encoding, cause the computer to perform any of the above-mentioned method embodiments for video decoding and / or encoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Other features, properties, and various advantages of the disclosed subject matter will be more apparent from the following detailed description and accompanying drawings.

[0040] Figure 1A A schematic diagram showing an exemplary subset of intra prediction direction modes.

[0041] Figure 1B A diagram of exemplary intra prediction directions is shown.

[0042] Figure 2 A schematic diagram showing spatial merging candidates of a current block and its surroundings for motion vector prediction in one example is shown.

[0043] Figure 3 A schematic diagram showing a simplified block diagram of a communication system according to an example embodiment.

[0044] Figure 4 A schematic diagram showing a simplified block diagram of a communication system according to an example embodiment.

[0045] Figure 5 A schematic diagram showing a simplified block diagram of a video decoder according to an example embodiment is shown.

[0046] Figure 6 A schematic diagram showing a simplified block diagram of a video encoder according to an example embodiment is shown.

[0047] Figure 7 A block diagram of a video encoder according to another example embodiment is shown.

[0048] Figure 8 A block diagram of a video decoder according to another example embodiment is shown.

[0049] Fig. 9 A scheme of coding block partitioning according to an example embodiment of the present disclosure is shown.

[0050] Fig.10 Another scheme of coding block partitioning according to an example embodiment of the present disclosure is shown.

[0051] Fig.11 Another scheme of coding block partitioning according to an example embodiment of the present disclosure is shown.

[0052] Fig.12 Another scheme of coding block partitioning according to an example embodiment of the present disclosure is shown.

[0053] Fig.13 A scheme for partitioning a coding block into a plurality of transform blocks and an encoding order of the transform blocks according to an example embodiment of the present disclosure is shown.

[0054] Fig.14 Another scheme for partitioning a coding block into a plurality of transform blocks and a coding order of the transform blocks according to an example embodiment of the present disclosure is shown.

[0055] Fig.15 Another scheme for partitioning a coding block into multiple transform blocks according to an example embodiment of the present disclosure is shown.

[0056] Fig.16 A flow chart according to an example embodiment of the present disclosure is shown.

[0057] Fig.17 A schematic diagram of a computer system according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Figure 3 3 is a simplified block diagram of a communication system (300) according to an embodiment disclosed in the present application. The communication system (300) includes a plurality of terminal devices, which can communicate with each other through, for example, a network (350). For example, the communication system (300) includes a first terminal device (310) and a second terminal device (320) interconnected through the network (350). Figure 3 In an embodiment of the present invention, the first terminal device (310) and the second terminal device (320) perform unidirectional data transmission. For example, the first terminal device (310) may encode video data (e.g., a video picture stream collected by the first terminal device (310)) for transmission to the second terminal device (320) via the network (350). The encoded video data is transmitted in the form of one or more encoded video code streams. The second terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video data, and display the video picture based on the restored video data. Unidirectional data transmission is more common in applications such as media services.

[0059] In another embodiment, the communication system (300) includes a third terminal device (330) and a fourth terminal device (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during a video conference. For bidirectional data transmission, each of the third terminal device (330) and the fourth terminal device (340) can encode video data (e.g., a video picture stream collected by the terminal device) to transmit to the other terminal device of the third terminal device (330) and the fourth terminal device (340) through a network (350). Each of the third terminal device (330) and the fourth terminal device (340) can also receive the encoded video data transmitted by the other terminal device of the third terminal device (330) and the fourth terminal device (340), and can decode the encoded video data to restore the video data, and can display the video picture on an accessible display device based on the restored video data.

[0060] exist Figure 3 In the embodiment of the present invention, the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340) may be servers, personal computers and smart phones, but the scope of application of the basic principles disclosed in the present application is not limited thereto. The embodiments disclosed in the present application are applicable to desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. The network (350) represents any number or type of network that transmits encoded video data between the first terminal device (310), the second terminal device (320), the third terminal device (330) and the fourth terminal device (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data in circuit switching, packet switching and / or other types of channels. The network may include a telecommunications network, a local area network, a wide area network and / or the Internet. For the purposes of the present application, unless otherwise explicitly explained in this document, the architecture and topology of the network (350) may be irrelevant to the operation disclosed in the present application.

[0061] As an example, Figure 4 The placement of the video encoder and the video decoder in a video streaming environment is shown. The subject matter disclosed in the present application is equally applicable to other video applications, including, for example, video conferencing, digital TV broadcasting, games, virtual reality, compressed video storage on digital media including CDs, DVDs, memory sticks, etc.

[0062] The video streaming system may include an acquisition subsystem (413), which may include a video source (401), such as a digital camera, to create, for example, an uncompressed video picture or image stream (402). In an embodiment, the video picture stream (402) includes samples recorded by the digital camera of the video source 401. Compared to the encoded video data (404) (or the encoded video bitstream), the uncompressed video picture stream (402) is depicted as a thick line to emphasize the high data volume of the video picture stream, and the video picture stream (402) can be processed by an electronic device (420), which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the uncompressed video picture stream (402), the encoded video data (404) (or the encoded video bitstream (404)) is depicted as a thin line to emphasize the lower amount of data of the encoded video data (404) (or the encoded video bitstream (404)), which can be stored on the streaming server (405) for future use or directly used by a downstream video device (not shown). One or more streaming client subsystems, such as Figure 4 The client subsystem (406) and the client subsystem (408) in the video transmission system can access the streaming server (405) to retrieve the copy (407) and the copy (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in the electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an uncompressed output video picture stream (411) that can be presented on a display (412) (e.g., a display screen) or another presentation device (not depicted). The video decoder 410 can be configured to perform some or all of the various functions described in the present disclosure. In some streaming transmission systems, the encoded video data (404), the video data (407), and the video data (409) (e.g., a video bitstream) can be encoded according to certain video encoding / compression standards. Examples of these standards include ITU-T H.265. In an embodiment, the video coding standard under development is informally referred to as Versatile Video Coding (VVC), and the present application can be used in the context of the VVC standard and other video coding standards.

[0063] It should be noted that the electronic device (420) and the electronic device (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0064] Figure 5 is a block diagram of a video decoder (510) according to an embodiment disclosed below in the present application. The video decoder (510) may be provided in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used to replace Figure 4 A video decoder (410) of an embodiment.

[0065] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one encoded video sequence is decoded at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. Each video sequence may be associated with multiple video frames or images. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data or a streaming source transmitting the encoded video data. The receiver (531) may receive the encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective processing circuits (not shown). The receiver (531) may separate the encoded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be configured between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be disposed outside the video decoder (510) and separated from the video decoder (510) (not shown). In other applications, a buffer memory (not shown) is disposed outside the video decoder (510) to, for example, prevent network jitter, and another buffer memory (515) may exist inside the video decoder (510) to, for example, handle broadcast timing. When the receiver (531) receives data from a storage / forward device with sufficient bandwidth and controllability or from an isochronous network, it may not be necessary to configure the buffer memory (515), or the buffer memory may be made smaller. Of course, in order to be used on a service packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and the buffer memory may be relatively large. Such a buffer memory may have an adaptive size and may be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (510).

[0066] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the encoded video sequence. The types of symbols include information used to manage the operation of the video decoder (510) and potential information used to control a display device such as a display device (512) (e.g., a display screen), which may or may not be part of the electronic device (530) but may be coupled to the electronic device (530), such as Figure 5 As shown in . The control information for the display device may be a parameter set fragment (not indicated) of Supplemental Enhancement Information (SEI message) or Video Usability Information (VUI). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy encoding of the encoded video sequence may be performed according to a video coding technique or standard, and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (520) may extract a subgroup parameter set of at least one subgroup of the subgroups of pixels in the video decoder from the encoded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), and the like. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients (eg, transform coefficients), quantizer parameter values, motion vectors, and so on.

[0067] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515), thereby creating symbols (521).

[0068] Depending on the type of coded video picture or portion of coded video picture (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors, the reconstruction of the symbol (521) may involve multiple different processing or functional units. Which units are involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (520) and the multiple processing or functional units described below is not described.

[0069] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical embodiment operating under commercial constraints, many of these functional units interact closely with each other and can be integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, the conceptual subdivision of functional units is adopted in the following disclosure.

[0070] The first unit may include a sealer / inverse transform unit (551). The sealer / inverse transform unit (551) receives quantized transform coefficients as symbols (521) and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc. The sealer / inverse transform unit (551) may output a block including sample values, which may be input into the aggregator (555).

[0071] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to an intra-coded block; for example, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses reconstructed surrounding block information stored in a current picture buffer (558) to generate surrounding blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) adds the prediction information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0072] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to an inter-frame coded and potentially motion compensated block. In this case, the motion compensated prediction unit (553) may access the reference picture memory (557) to extract samples for inter-frame picture prediction. After the extracted samples are motion compensated according to the symbols (521), these samples may be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) (the output of unit 551 is called residual samples or residual signal), thereby generating output sample information. The acquisition of prediction samples by the motion compensated prediction unit (553) from the address in the reference picture memory (557) may be controlled by a motion vector, and the motion vector is provided to the motion compensated prediction unit (553) in the form of the symbols (521), for example, including X, Y components (displacement) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory (557) when using sub-sample accurate motion vectors, motion compensation may also be associated with a motion vector prediction mechanism, etc.

[0073] The output samples of the aggregator (555) may be employed by various loop filtering techniques in a loop filter unit (554). The video compression techniques may include in-loop filter techniques controlled by parameters included in an encoded video sequence (also referred to as an encoded video bitstream) and available to the loop filter unit (556) as symbols (521) from the parser (520). However, in other embodiments, the video compression techniques may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, as well as to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in further detail below.

[0074] The output of the loop filter unit (556) may be a sample stream that may be output to a display device (512) and stored in a reference picture memory (557) for subsequent inter-picture prediction.

[0075] Once fully reconstructed, certain coded pictures may be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed, and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and a new current picture buffer may be reallocated before starting to reconstruct a subsequent coded picture.

[0076] The video decoder (510) may perform decoding operations according to a predetermined video compression technique employed, for example, in the ITU-T H.265 standard. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile recorded in the video compression technique or standard. Specifically, the profile may select certain tools from all the tools available in the video compression technique or standard as the only tools available for use under the profile. For compliance, the complexity of the encoded video sequence is within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum picture size, the maximum frame rate, the maximum reconstruction sampling rate (measured in, for example, mega samples per second), the maximum reference picture size, etc. In some cases, the limits set by the hierarchy may be further defined by the Hypothetical Reference Decoder (HRD) specification and metadata of the HRD buffer management signaled in the encoded video sequence.

[0077] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0078] Figure 6 6 is a block diagram of a video encoder (603) according to an embodiment disclosed in the present application. The video encoder (603) is provided in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used to replace Figure 4 A video encoder (403) in an embodiment.

[0079] The video encoder (603) can be used to obtain the video source (601) (not Figure 6 In another embodiment, the video source (601) may be implemented as a part of the electronic device (620) to receive video samples, and the video source may capture video images to be encoded by the video encoder (603). In another embodiment, the video source (601) may be implemented as a part of the electronic device (620).

[0080] The video source (601) may provide a source video sequence in the form of a digital video sample stream to be encoded by the video encoder (603), wherein the digital video sample stream may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, XYZ, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures or images that are given motion when viewed sequentially. The picture itself may be constructed as a spatial pixel array, where each pixel may include one or more samples depending on the sampling structure, color space, etc. being used. The relationship between pixels and samples may be easily understood by those skilled in the art. The following description focuses on samples.

[0081] According to an embodiment, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to these units. For the sake of brevity, couplings are not shown in the figure. The parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, lambda value of rate-distortion optimization technology, etc.), picture size, group of pictures (GOP) layout, maximum allowed motion vector search range, etc. The controller (650) can be used to have other suitable functions that are related to the video encoder (503) optimized for a certain system design.

[0082] In some embodiments, the video encoder (603) operates in a coding loop. As a brief description, in embodiments, the coding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how the (remote) decoder creates sample data, even if the embedded decoder 633 processes the encoded video stream through the source encoder 630 without entropy coding (because in the video compression technology considered in this application, any compression between the symbols and the encoded video code stream is lossless). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-accurate results that are independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction part of the encoder are exactly the same as the sample values ​​that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the drift that occurs when synchronization cannot be maintained, eg due to channel errors) is used to improve the coding quality.

[0083] The operation of the "local" decoder (633) can be combined with the above Figure 5 The "remote" decoder described in detail for the video decoder (510) is identical. However, additional brief reference is made to Figure 5 , when symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into an encoded video sequence, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the encoder's local decoder (633).

[0084] At this point, it can be observed that any decoder technology other than the parsing / entropy decoding present in the decoder must also be present in the corresponding encoder in substantially the same functional form. For this reason, the present application sometimes focuses on the decoder operation, which is related to the decoding part of the encoder. The description of the encoder technology can be simplified because the encoder technology is mutually inverse to the decoder technology described comprehensively. The encoder is described in more detail only in certain areas or aspects below.

[0085] During operation, in some embodiments, the source encoder (630) may perform motion compensated predictive coding. The motion compensated predictive coding predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this manner, the encoding engine (632) encodes the differences (residues) between blocks of pixels in color channels of the input picture and blocks of pixels of a reference picture that may be selected as a prediction reference for the input picture. The term "residue" and its adjective form "residual" may be used interchangeably.

[0086] The local video decoder (633) may decode the encoded video data of the picture that may be designated as the reference picture based on the symbol created by the source encoder (630). The operation of the encoding engine (632) may be a lossy process. When the encoded video data is available at the video decoder ( Figure 6 When the video encoder (603) is decoded at a local (not shown), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process that may be performed by the video decoder on the reference picture and may cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may store a copy of the reconstructed reference picture locally that has common content (absent transmission errors) with the reconstructed reference picture to be obtained by the far-end (remote) video decoder.

[0087] The predictor (635) may perform a prediction search for the encoding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate pixel-by-pixel based on sample blocks to find a suitable prediction reference. In some cases, based on the search results obtained by the predictor (635), it may be determined that the input picture may have prediction references taken from a plurality of reference pictures stored in the reference picture memory (634).

[0088] The controller (650) may manage encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0089] The outputs of all the above functional units may be entropy encoded in an entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc., thereby converting the symbols into a coded video sequence.

[0090] The transmitter (640) may buffer the encoded video sequence created by the entropy encoder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device where the encoded video data will be stored. The transmitter (640) may combine the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or ancillary data streams (source not shown).

[0091] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a certain coded picture type to each coded picture, but this may affect the coding techniques that can be applied to the corresponding picture. For example, a picture may generally be assigned to any of the following picture types:

[0092] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I pictures and their corresponding applications and features.

[0093] A predictive picture (P picture) may be a picture that can be encoded and decoded using intra prediction or inter prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0094] Bidirectional predictive pictures (B pictures), which can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstruction of a single block.

[0095] The source picture may typically be spatially subdivided into blocks of samples (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples) and encoded block by block. These blocks may be predictively encoded with reference to other (already encoded) blocks, which are determined according to the coding allocation applied to the corresponding picture of the block. For example, blocks of an I picture may be non-predictively encoded, or the blocks may be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same picture. Blocks of pixels of a P picture may be predictively encoded by spatial prediction with reference to one previously encoded reference picture or by temporal prediction. Blocks of a B picture may be predictively encoded by spatial prediction with reference to one or two previously encoded reference pictures or by temporal prediction. For other purposes, source pictures or intermediately processed pictures may be subdivided into other types of blocks. The division of coding blocks and other types of blocks may or may not follow the same approach, as described in further detail below.

[0096] The video encoder (603) may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0097] In an embodiment, the transmitter (640) may transmit additional data when transmitting the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0098] The captured video may be taken as a plurality of source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often simplified to intra-prediction) exploits spatial correlations in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an embodiment, a particular picture being encoded / decoded is divided into blocks, and the particular picture being encoded / decoded is referred to as the current picture. When a block in the current picture is similar to a reference block in a reference picture that was previously encoded in the video and is still buffered, the block in the current picture may be encoded by a vector called a motion vector. The motion vector points to a reference block in a reference picture, and in the case where multiple reference pictures are used, the motion vector may have a third dimension that identifies the reference picture.

[0099] In some embodiments, bidirectional prediction techniques may be used for inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture that are both before the current picture in the video in decoding order (but may be in the past or future in display order, respectively). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block may be jointly predicted by a combination of the first reference block and the second reference block.

[0100] In addition, merge mode technology can be used in inter-picture prediction to improve coding efficiency.

[0101] According to some embodiments disclosed in the present application, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a video picture sequence is divided into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three parallel coding tree blocks (CTBs): a luminance CTB and two chrominance CTBs. Furthermore, each CTU can be split into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be split into a 64×64 pixel CU, or four 32×32 pixel CUs. Each of the one or more 32×32 blocks can be further divided into four 16×16 pixel CUs. In one embodiment, each CU can be analyzed during encoding to determine the prediction type for the CU among various prediction types, such as an inter-prediction type or an intra-prediction type. In addition, depending on temporal and / or spatial predictability, the CU is split into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in the encoding (encoding / decoding) is performed in units of prediction blocks. The partitioning of the CU into PUs (or PBs of different color channels) may be performed in various spatial modes. For example, a luminance or chrominance PB may include a value matrix (e.g., luminance values) of samples, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, and 16x8 samples, etc.

[0102] Figure 7is a diagram of a video encoder (703) according to another exemplary embodiment disclosed in the present application. The video encoder (703) is used to receive a processed block (e.g., a prediction block) of sample values ​​within a current video picture in a video picture sequence, and encode the processed block into an encoded picture that is part of an encoded video sequence. In this embodiment, the video encoder (703) is used to replace Figure 4 A video encoder (303) in an embodiment.

[0103] In one embodiment, a video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8×8 samples. The video encoder (703) then uses, for example, rate-distortion optimization (RDO) to determine whether to use intra mode, inter mode, or bidirectional prediction mode to encode the processing block. When it is determined that the processing block is encoded in intra mode, the video encoder (703) may use intra prediction techniques to encode the processing block into an encoded picture; and when it is determined that the processing block is encoded in inter mode or bidirectional prediction mode, the video encoder (703) may use inter prediction or bidirectional prediction techniques to encode the processing block into an encoded picture, respectively. In some exemplary embodiments, merge mode may be a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the aid of an encoded motion vector component external to the predictor. In certain other exemplary embodiments, there may be a motion vector component applicable to the subject block. Therefore, the video encoder (703) includes providing a method for encoding the processing block in the following manner: Figure 7 Components explicitly shown in the example, such as a mode decision module (not shown) for determining a prediction mode for a processing block.

[0104] exist Figure 7 In an embodiment of the present invention, the video encoder (703) includes: Figure 7 An inter-frame encoder (730), an intra-frame encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721) and an entropy encoder (725) coupled together are shown in an exemplary arrangement.

[0105] The inter-frame encoder (730) is used to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture in display order), generate inter-frame prediction information (e.g., redundant information description according to an inter-frame coding technique, motion vectors, merge mode information), and calculate an inter-frame prediction result (e.g., a prediction block) based on the inter-frame prediction information using any suitable technique. In some embodiments, the reference picture is embedded in Figure 6 The decoding unit 633 in the example encoder 620 (such as Figure 7 The residual decoder 728 is shown in detail below), which decodes the decoded reference picture based on the encoded video information.

[0106] The intra-frame encoder (722) is used to receive samples of a current block (e.g., a processing block), compare the block with an encoded block in the same picture in some cases, generate quantization coefficients after transformation, and in some cases also generate intra-frame prediction information (e.g., intra-frame prediction direction information according to one or more intra-frame coding techniques). The intra-frame encoder (722) calculates an intra-frame prediction result (e.g., a prediction block) based on the intra-frame prediction information and a reference block in the same picture.

[0107] The general controller (721) is used to determine general control data and control other components of the video encoder (703) based on the general control data. In an embodiment, the general controller (721) determines the prediction mode of the block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and add the intra prediction information to the bitstream; and when the prediction mode for the block is inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and add the inter prediction information to the bitstream.

[0108] The residual calculator (723) is used to calculate the difference (residual data) between the received block and the prediction result of the block selected from the intra-frame encoder (722) or the inter-frame encoder (730). The residual encoder (724) is used to encode the residual data to generate a transform coefficient. In an embodiment, the residual encoder (724) is used to convert the residual data from the time domain to the frequency domain to generate a transform coefficient. The transform coefficient is then quantized to obtain a quantized transform coefficient. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is used to perform an inverse transform and generate decoded residual data. The decoded residual data can be used appropriately by the intra-frame encoder (722) and the inter-frame encoder (730). For example, the inter-frame encoder (730) can generate a decoded block based on the decoded residual data and the inter-frame prediction information, and the intra-frame encoder (722) can generate a decoded block based on the decoded residual data and the intra-frame prediction information. The decoded blocks are appropriately processed to generate a decoded picture, and the decoded picture may be buffered in a memory circuit (not shown) and used as a reference picture.

[0109] The entropy encoder (725) is used to format the code stream to produce encoded blocks and perform entropy encoding. The entropy encoder (725) is configured to include various information in the code stream. In an embodiment, the entropy encoder (725) is used to obtain general control data, selected prediction information (such as intra-frame prediction information or inter-frame prediction information), residual information, and other suitable information in the code stream. It should be noted that when the block is encoded in the inter-frame mode or the merge sub-mode of the bidirectional prediction mode, there is no residual information.

[0110] Figure 8 FIG. 8 is a diagram of a video decoder (810) according to another embodiment disclosed in the present application. The video decoder (810) is used to receive an encoded image as part of an encoded video sequence and decode the encoded image to generate a reconstructed picture. In an embodiment, the video decoder (810) is used to replace Figure 4 A video decoder (410) of an embodiment.

[0111] exist Figure 8 In one embodiment, the video decoder (810) includes: Figure 8 The schematic arrangement shown in FIG. 8 is an entropy decoder ( 871 ), an inter-frame decoder ( 880 ), a residual decoder ( 873 ), a reconstruction module ( 874 ) and an intra-frame decoder ( 872 ) coupled together.

[0112] The entropy decoder (871) may be used to reconstruct certain symbols from the encoded picture, which represent syntax elements that constitute the encoded picture. Such symbols may include, for example, a mode for encoding the block (e.g., intra mode, inter mode, bidirectional prediction mode, merge submode, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that may identify certain samples or metadata for prediction by the intra decoder (872) or the inter decoder (880), residual information in the form of, for example, quantized transform coefficients, and the like. In an embodiment, when the prediction mode is inter or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0113] The inter-frame decoder (880) is used to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0114] The intra-frame decoder (872) is used to receive intra-frame prediction information and generate a prediction result based on the intra-frame prediction information.

[0115] The residual decoder (873) is used to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use some control information (to obtain the quantizer parameter QP), which may be provided by the entropy decoder (871) (the data path is not indicated because this is only low-data-volume control information).

[0116] The reconstruction module (874) is used to combine the residual output by the residual decoder (873) with the prediction result (which can be output by the inter-frame prediction module or the intra-frame prediction module) in the spatial domain to form a reconstructed block, and the reconstructed block forms a part of the reconstructed picture, and the reconstructed picture can be used as a part of the reconstructed video. It should be noted that other suitable operations such as deblocking operations can be performed to improve visual quality.

[0117] It should be noted that the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using any suitable technology. In an embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), video encoder (603), and video encoder (703), as well as the video decoder (410), video decoder (510), and video decoder (810) may be implemented using one or more processors that execute software instructions.

[0118] Turning to encoding the block partitions, and in some example embodiments, a predetermined pattern may be applied. Fig. 9 As shown, an example 4-way partition tree starting from a first predetermined level (e.g., 64×64 block level) down to a second predefined level (e.g., 4×4 level) may be employed. For example, a basic block may be restricted to four partitioning options indicated by 902, 904, 906, and 908, where the partition designated as R allows for recursive partitioning, Fig. 9 The same partition tree shown may be repeated at lower scales until the lowest level (e.g., a 4×4 level). In some embodiments, additional restrictions may be applied to Fig. 9 The partition scheme of Fig. 9 In an embodiment of , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be allowed to be recursive, while square partitions may be allowed to be recursive. If necessary, follow Fig. 9 The recursive partitioning of generates the final set of coded blocks. Such a scheme can be applied to one or more color channels.

[0119] Fig.10 Another example predefined partitioning mode that allows recursive partitioning to form a partition tree is shown. Fig.10 As shown, an example 10-way partition structure or pattern may be predefined. A root block may start at a predefined level (eg, from a 128x128 level, or a 64x64 level). Fig.10 Example partition structures include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Fig.10 The second row 1002, 1004, 1006, and 1008 indicate a partition type with three sub-partitions, which can be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 can be referred to as a left T-type, a top T-type, a right T-type, and a bottom T-type. In some embodiments, Fig.10 The rectangular partition does not allow further subdivision. The coding tree depth can be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., 128×128 block) can be set to 0, and the coding tree depth of the root block can be set to 0. Fig.10 After the pattern is further split once, the coding tree depth is increased by 1. In some embodiments, all square partitions in 1010 may be allowed to be divided according to Fig.10 In other words, for square partitions with patterns 1002, 1004, 1006, and 1008, recursive partitioning may not be allowed. If necessary, follow Fig.10 The recursive partitioning of generates the final set of coded blocks. Such a scheme can be applied to one or more color channels.

[0120] After dividing or partitioning the basic blocks according to any of the above partitioning processes or other processes, the final partition or coding block set can also be obtained. Each of these partitions can be in one of various partition levels. Each partition can be referred to as a coding block (CB). For the various example partitioning implementation schemes above, each generated CB can be of any allowed size and partition level. They are called coding blocks because they can form units to which some basic encoding / decoding decisions can be made, and the encoding / decoding parameters can be optimized, determined and signaled in the encoded video bitstream. The highest level in the final partition represents the depth of the coding block partition tree. The coding block can be a luminance coding block or a chrominance coding block.

[0121] In some other example embodiments, a quadtree structure may be used to recursively partition the base luma and chroma blocks into coding units. Such a partitioning structure may be referred to as a coding tree unit (CTU), which is partitioned into coding units (CUs) using a quadtree structure to adapt the partitioning to the various local characteristics of the base CTU. In such embodiments, an implicit quadtree partitioning is performed at a picture boundary so that the block will remain quadtree partitioned until the size fits the picture boundary. The term CU is used to collectively refer to the unit of luma and chroma coding blocks (CBs).

[0122] In some embodiments, CB can be further partitioned. For example, for the purpose of intra-frame or inter-frame prediction in the encoding and decoding process, CB can be further partitioned into multiple prediction blocks (PB). In other words, CB can be further divided into different sub-partitions, in which a separate prediction decision / configuration can be performed. In parallel, for the purpose of describing the level of the transformation or inverse transformation of the video data, CB can be further partitioned into multiple transform blocks (TB). The partition schemes of CB to PB and TB can be the same or different. For example, each partition scheme can be performed using its own process based on various characteristics of the video data, for example. In some example embodiments, PB and TB partition schemes can be independent. In some other example embodiments, PB and TB partition schemes and boundaries can be related. In some embodiments, for example, TB can be partitioned after PB partition, in particular, each PB is further partitioned into one or more TBs after being determined according to the partition of the coding block. For example, in some embodiments, PB can be divided into one, two, four or other number of TBs.

[0123] In some embodiments, in order to partition the basic block into coding blocks and further partition into prediction blocks and / or transform blocks, the luminance channel and the chrominance channel can be processed differently. For example, in some embodiments, the coding block of the luminance channel can be allowed to be partitioned into prediction blocks and / or transform blocks, while the coding block of the chrominance channel is not allowed to be partitioned into prediction blocks and / or transform blocks. In such embodiments, the transformation and / or prediction of the luminance block can therefore be performed only at the coding block level. For another example, the minimum transform block size of the luminance channel and the chrominance channel can be different, for example, the coding block of the luminance channel can be allowed to be partitioned into a transform block and / or prediction block that is smaller than the chrominance channel. For another example, the maximum depth of partitioning the coding block into transform blocks and / or prediction blocks can be different between the luminance channel and the chrominance channel, for example, the coding block of the luminance channel can be allowed to be partitioned into a transform block and / or prediction block that is deeper than the chrominance channel. For a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes, which may be represented by recursive partitioning down to up to 2 levels, and may allow transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4×4 to 64×64. However, for chroma blocks, only the largest possible transform block may be allowed to be specified for the luma block.

[0124] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of partitioning the PBs may depend on whether the PBs are intra-coded or inter-coded.

[0125] In various example schemes, partitioning of a coding block (or prediction block) into transform blocks may be implemented, including but not limited to recursively or non-recursively quadtree partitioning and predefined pattern partitioning, and additional consideration of transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks may be at different partitioning levels, may not have the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio).

[0126] In some embodiments, a coding partition tree scheme or structure can be used. The coding partition tree scheme for luminance and chrominance channels does not need to be the same. In other words, luminance and chrominance channels can have a separate coding tree structure. In addition, whether the luminance and chrominance channels use the same or different coding partition tree structures and the actual coding partition tree structure to be used can depend on whether the slice being encoded is a P, B or I slice. For example, for an I slice, the chrominance channel and the luminance channel can have a separate coding partition tree structure or coding partition tree structure mode, while for a P or B slice, the luminance and chrominance channels can share the same coding partition tree scheme. When a separate coding partition tree structure or mode is applied, the luminance channel can be partitioned into CBs by a coding partition tree structure, and the chrominance channel can be partitioned into chrominance CBs by another coding partition tree structure.

[0127] The specific example implementation scheme of coding block and transform block partition is described below. In such example implementation scheme, the recursive quadtree partition described above can be used to partition the basic coding block into coding blocks. At each level, whether the further quadtree partition of a specific partition should continue can be determined by the local video data characteristics. The generated CB can be in various quadtree partition levels of various sizes. The decision whether to use inter-picture (time) or intra-picture (space) prediction to encode the picture area can be made at the CB level (or CU, for all three color channel channels). Each CB can be further partitioned into one, two, four or other number of PBs according to the PB partition type. In a PB, the same prediction process can be applied, and the relevant information is sent to the decoder on the basis of the PB. After obtaining the residual block by applying the prediction process based on the PB partition type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this specific implementation scheme, the CB or TB can be but not necessarily limited to a square. In addition, in this specific example, the PB can be a square or rectangular shape for inter-frame prediction and can be only a square for intra-frame prediction. The coding block can be further partitioned into, for example, four square-shaped TBs. Each TB can be further recursively partitioned (using quadtree partitioning) into smaller TBs, called Residual Quad-Tree (RQT).

[0128] Another specific example for partitioning a basic coding block into CBs and other PBs and / or TBs is described below. For example, a quadtree with nested multi-type trees using binary and ternary segmentation structures, rather than using a quadtree such as Fig.10The multiple partition unit types shown in . The separation of CB, PB and TB concepts (i.e., partitioning CB into PB and / or TB, and partitioning PB into TB) can be abandoned, except when a CB with a size that is too large for the maximum transform length is required, where such CB may need to be further partitioned. This example partitioning scheme can be designed to support more flexibility in the shape of CB partitions, so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CU can have a square or rectangular shape. Specifically, the coding tree block (CTB) can first be partitioned by a quadtree structure. Then, the quadtree leaf nodes can be further partitioned by a multi-type tree structure. Fig.11 An example of a multi-type tree structure is shown in FIG. Fig.11 The example multi-type tree structure includes four types of segmentation, which are called vertical binary segmentation (SPLIT_BT_VER) (1102), horizontal binary segmentation (SPLIT_BT_HOR) (1104), vertical ternary segmentation (SPLIT_TT_VER) (1106) and horizontal ternary segmentation (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaves of the multi-type tree. In this example embodiment, unless the CB is too large for the maximum transform length, the segment is used for prediction and transform processing without any further partitioning. This means that in most cases, CB, PB and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is less than the width or height of the color component of the CB.

[0129] Fig.12 An example of a quadtree with a nested multi-type tree coding block structure for block partitioning of a CTB is shown in FIG. Fig.12 The CTB 1200 is shown to be partitioned by a quadtree into four square partitions 1202, 1204, 1206, and 1208. Fig.11 The multi-type tree structure of the quadtree partition is split into each partition. Fig.12 In the example of , partition 1204 is not further split. Partitions 1202 and 1208 each use another quadtree split. For partition 1202, the upper left, upper right, lower left, and lower right partitions of the second level quadtree split use the third level of quadtree split, Fig.11 1104, undivided and Fig.11 1108. Partition 1208 uses another quadtree partition, and the upper left, upper right, lower left and lower right partitions of the second level quadtree partition use Fig.11 The third level of 1106 is split, unsplit, unsplit and Fig.111104. Two of the sub-partitions of the third-level upper left partition 1208 are further divided according to 1104 and 1108. Partition 1206 is divided according to Fig.11 The second level partitioning mode of 1102 is divided into two partitions, the two partitions are based on Fig.11 1108 and 1102 are further divided into the third level. Fig.11 At 1104, a fourth level of segmentation is further applied to one of them.

[0130] For the specific example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different than luma at, for example, 32x32. When the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block may be automatically split in the horizontal and / or vertical direction to meet the transform size constraints in that direction.

[0131] In a specific example for partitioning a base coding block into the above CBs, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P and B slices, the luma and chroma CTBs in one CTU can share the same coding tree structure. For example, for an I slice, luma and chroma can have separate coding block tree structures. When the separated block tree mode is applied, the luma CTB is partitioned into CBs through one coding tree structure, and the chroma CTB is partitioned into chroma CBs through another coding tree structure. This means that a CU in an I slice can consist of coding blocks of a luma component or coding blocks of two chroma components, and a CU in a P or B slice always consists of coding blocks of all three color components unless the video is monochrome.

[0132] Example implementations for partitioning a coding block or prediction block into transform blocks and the encoding order of transform blocks are described in further detail below. In some example implementations, transform partitioning can support multiple shapes, for example, 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1 transform blocks, where transform block sizes range from, for example, 4×4 to 64×64. In some implementations, if the coding block is less than or equal to 64×64, transform block partitioning can be applied only to the luma component, so that for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, then the luma and chroma coding blocks can be implicitly split into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform blocks, respectively.

[0133] In some example embodiments, for both intra-coded blocks and inter-coded blocks, the coded blocks may be further partitioned into multiple transform blocks with up to a predetermined number of partition depths (e.g., 2 levels). The transform block partition depth and size may be related. An example mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 below.

[0134] Table 1: Change partition size settings

[0135]

[0136] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partitioning can create four 1:1 square sub-transform blocks. The transform partitioning can stop at 4×4, for example. In this way, the transform size of the current depth 4×4 corresponds to the same size 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partitioning will create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform partitioning will create two 1:2 / 2:1 sub-transform blocks.

[0137] In some example embodiments, for the luma component of an intra-coded block, additional restrictions may be applied. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have the same size. For example, for a 32×16 coding block, a level 1 transform partition creates two 16×16 sub-transform blocks and a level 2 transform partition creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform unit sizes equal. Fig.13 1 shows an example of transform block partitioning for intra-coding square blocks as shown in Table 1, and the coding order illustrated by arrows. Specifically, 1302 shows a square coding block. In 1304, a first level partitioning into 4 equal-sized transform blocks according to Table 1 is shown, where the coding order is indicated by arrows. In 1306, a second level partitioning of all first level equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown, where the coding order is indicated by arrows.

[0138] In some example embodiments, the above restrictions on intra-coding may not apply to the luminance component of an inter-coded block. For example, after the first level of transform partitioning, any of the sub-transform blocks may be further partitioned one level independently. Thus, the resulting transform blocks may or may not have the same size. Fig.14 An example of dividing an inter-frame coding block into transform blocks is shown in FIG. 1 , and the transform blocks have a coding order. Fig.14In the example of FIG. 1402, an inter-coded block 1402 is split into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, as shown in 1404, only one of the four transform blocks (not all) is further split into four sub-transform blocks, resulting in a total of 7 transform blocks with two different sizes. The example coding order of these 7 transform blocks is as follows: Fig.14 The arrow in 1404 is shown.

[0139] In some example embodiments, for one or more chroma components, some additional restrictions on transform blocks may be applied. For example, for one or more chroma components, the transform block size may be as large as the encoding block size, but not smaller than a predefined size, such as 8×8.

[0140] In some other example embodiments, for coding blocks with width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly split into multiples of min(W, 64)×min(H, 64) and min(W, 32)×min(H, 32) transform units, respectively.

[0141] Fig.15 Another optional example scheme for partitioning a coding block or prediction block into transform blocks is further shown. Fig.15 As shown, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coding block according to the transform type of the coding block. Fig.15 In the specific example shown in , one of 6 example partition types can be applied to split the coding block into various numbers of transform blocks. Such a scheme can be applied to either the coding block or the prediction block.

[0142] In more detail, Fig.15 The partition scheme is as follows Fig.15 Any given transform type shown provides up to 6 partition types. In this scheme, a transform type can be assigned to each coding block or prediction block based on, for example, rate-distortion cost. In an example, the partition type assigned to a coding block or prediction block can be determined based on the transform partition type of the coding block or prediction block. A specific partition type can correspond to a transform block partition size and mode (or partition type), such as Fig.15 As shown in the 6 partition types illustrated in FIG. The correspondence between various transform types and various partition types can be predefined. The corresponding examples are shown below, where the capitalized mark indicates the transform type that can be assigned to the coding block or prediction block based on the rate-distortion cost:

[0143] PARTITION_NONE: Allocate a transform size equal to the block size.

[0144] PARTITION_SPLIT: The allocated transform size is 1 / 2 of the block size in width and 1 / 2 of the block size in height.

[0145] PARTITION_HORZ: Allocate a transform size with the same width as the block size and 1 / 2 the height of the block size.

[0146] PARTITION_VERT: Allocates a transform size with 1 / 2 the width of the block size and the same height as the block size.

[0147] PARTITION_HORZ4: Allocates a transform size with the same width as the block size and 1 / 4 of the height of the block size.

[0148] PARTITION_VERT4: Allocates a transform size with 1 / 4 the width of the block size and the same height as the block size.

[0149] In the above example, if Fig.15 All partition types shown include a uniform transform size for the partition transform block. This is merely an example and not a limitation. In some other embodiments, a hybrid transform block size may be used for the partition transform blocks of a particular partition type (or mode).

[0150] Turning to some example embodiments of specific types of signaling and entropy coding and decoding of various blocks / units, for each intra-frame and inter-frame coded block / unit, a flag, namely a skip_txfm flag, can be signaled in the coded code stream, as shown in the example syntax of Table 2 below, and represented by the read_skip() function, which is used to retrieve these flags from the code stream. The flag can indicate whether the transform coefficients are all zero in the current coding unit. The skip_txfm flag can be referred to as a zero-skip flag. In some example embodiments, if the flag is signaled with a value such as 1, then for any coding block in the coding unit, another transform coefficient related syntax (e.g., EOB (end of block)) does not need to be signaled, and the zero skip flag can be derived as: a value or data structure predefined for and associated with a zero transform coefficient block. For inter-frame coded blocks, as shown in the example of Table 2, the flag can be signaled after the skip_mode flag, which indicates that the coding unit can be skipped for various reasons. When the skip_mode flag value is true, the coding unit should be skipped and no skip_txfm flag needs to be signaled, and the skip_txfm flag is inferred to be 1. Otherwise, if the skip_mode flag is false, more information about the coding block / unit will be included in the codestream, and the skip_txfm flag will be additionally signaled to indicate whether the coding block / unit is all zero.

[0151] Table 2: Skip patterns and skip syntax

[0152]

[0153]

[0154]

[0155] In this way, the skip_txfm flag can be used to indicate whether the block associated with the flag is all zero and whether it can be skipped. For example, if the flag is signaled as 1, then the block may contain all zero coefficients. Otherwise, if the flag is signaled as zero, then at least some of the coefficients in the block are non-zero. In some embodiments, the block may be divided or partitioned into transform blocks in various ways as described above. Each of the t transform blocks may be additionally associated with an EOB skip flag that may be signaled in the encoded bitstream. If the EOB skip flag for a particular transform block is signaled as 1, then the EOB for the transform block will not be included in the bitstream, and the transform block may be skipped because it does not have non-zero coefficients. However, when the EOB skip flag for the transform block is signaled as 0, then this indicates that there is at least one non-zero coefficient for the transform block, and indicates that the transform block cannot be skipped. In the following example embodiments, the focus is on signaling the skip_txfm flag and encoding it. For simplicity, the above skip_txfm flag may be referred to as a skip flag.

[0156] A context set for entropy coding of a skip_txfm flag (or zero skip flag) in a code stream may be designed. A context for encoding a skip_txfm flag for a specific current coding block may be selected from the context set. The selection of a context may be indicated by an index. Selection from a context set may depend on various factors. For example, the selection of a coding context for skip_txfm of the current block may depend on the skip_txfm flag value of at least one of the neighboring blocks of the current block. For example, at least one neighboring block may include an upper and / or left neighboring block of the current block. For example, a total of three different contexts may be selected. In a specific example implementation, if neither the upper or left neighboring block is encoded with a non-zero skip_txfm flag, then a context index value of 0 is used to encode the skip_txfm flag of the current block. If one of the upper and left neighboring blocks is encoded with a non-zero skip_txfm flag, then a context index value of 1 may be used to encode the skip_txfm flag of the current block. If both the upper and left neighboring blocks are encoded with non-zero skip_txfm flags, then the context index value 2 is used to encode the skip_txfm flag of the current block. This implementation is based on the statistical observation that the more neighboring blocks have all-zero coefficients, the more likely the current block is to have all-zero coefficients. An example process for selecting from a context set for encoding the current skip_txfm flag is shown in Table 3 below.

[0157] Table 3. Neighboring block related context derivation for the skip_txfm flag; cdf is given by TileSkipCdf[ctx]

[0158]

[0159]

[0160] The following additional example implementations focus on a scheme for further improving the entropy coding efficiency of the skip_txfm flag and some other flags in the bitstream by utilizing some statistical correlations between these flags and other flags within the current coding block or across coding blocks (e.g., between adjacent blocks). Specifically, the context for encoding these flags of the coding block is designed to depend on the prediction mode (intra-frame or inter-frame prediction mode) of the coding block and / or its adjacent coding blocks.

[0161] For example, some of these example embodiments may be based on the following observation: in addition to the correlation of the skip_txfm flag values ​​between neighboring blocks, the probability that the skip_txfm value of the current block is equal to 1 may also depend largely on whether the current block is an intra-coded block or an inter-coded block. By considering the prediction mode when determining the context in which the skip_txfm flag is encoded, higher coding efficiency may be obtained due to the possible correlation between the skip_txfm flag value and the prediction mode.

[0162] The various example embodiments described below may be used individually or in combination in any order. Further, each of these embodiments may be embodied as a part of an encoder and / or decoder, and may be implemented in hardware or software. For example, these embodiments may be hard-coded in a dedicated processing circuit (e.g., one or more integrated circuits). In another example, these embodiments may be implemented by one or more processors executing a program stored in a non-volatile computer-readable medium. In addition, the term "block" may refer to any unit of video information, and the term "block size" may refer to a block width or height, or a maximum value of width and height, or a minimum value of width and height, or an area size (width*height), or an aspect ratio of a block (width:height, or height:width).

[0163] In some general example implementations, multiple contexts may be designed to encode the skip_txfm flag of a current block of video in a bitstream. The skip_txfm flag used to signal a particular current block and the context for encoding it may depend on the prediction mode of the current block, such as whether the current block is intra-coded or inter-coded.

[0164] In some example embodiments, when the current block is an inter-coded block, the context for signaling the skip_txfm flag and encoding the flag may depend on the following two conditions: a) whether at least one neighboring block of the current block (e.g., the blocks above and / or to the left) is intra-coded; and b) the skip_txfm flag of at least one neighboring block of the current block (e.g., the neighboring blocks above or to the left).

[0165] In one specific example implementation using both the above and left neighboring blocks, if the current block is an inter-coded block, the context selection (context index value) implementation for encoding the above current skip_txfm flag based on the neighboring skip_txfm flag may be further modified according to the following equation:

[0166] above_skip_txfm=is_inter(above_mi)? above_mi→skip_txfm: 0 (1)

[0167] left_skip_txfm=is_inter(1eft_mi)? left_mi→skip_txfm: 0 (2)

[0168] context=above_skip_txfm+left_skip_txfm (3)

[0169] Where is_inter(left_mi) or is_inter(above_mi) returns a Boolean value indicating whether the left block or the above block is an inter-frame coded block, above_mi→skip_txfm represents the transform skip flag (skip_txfm) of the above block, and 1eft_mi→skip_txfm represents the transform skip flag of the left block. These adjacent block skip_txfm flags can be referred to as reference skip_txfm flags, which are used as references for generating "above_skip_txfm" and "above_skip_txfm", which are called prediction mode related adjacent skip_txfm flags. The context selection index can be the sum of the prediction mode related adjacent skip_txfm flags.

[0170] Following the above example, there may be three example contexts represented by contexts 0, 1, and 2. When the current block is an intra-predicted block, determining the context for encoding the current skip_txfm flag may follow the example process of Table 3 or other processes. However, when the current block is an inter-predicted block, the context selection may be modified according to the above equation so that the skip_txfm flag of the neighboring block considered in Table 3 is replaced by the prediction mode-dependent neighboring skip_txfm flag in the above equation. The prediction mode-dependent neighboring skip_txfm flag essentially represents the actual neighboring skip_txfm flag modified depending on the prediction mode of the neighboring block. In particular, if the neighboring block is an inter-coded block, the actual skip_txfm flag of the neighboring block will be maintained as the prediction mode-dependent skip_txfm flag, and otherwise, if the neighboring block is an intra-coded block, the prediction mode-dependent skip_txfm flag of the neighboring block will be set to 0 (as if the neighboring block is not an all-zero coefficient, regardless of whether this is actually true). In essence, the above embodiments exploit the observation that when one or more neighboring blocks are intra predicted, then it is more likely that the current block is not all non-zero.

[0171] The above equation is only used for the example of generating three different context indexes pointing to three different contexts. Other mathematical formulas can be used to make this distinction, and there can be more than three different contexts and corresponding context indexes.

[0172] In some other or further example embodiments, the context for signaling the skip_txfm flag and encoding it may depend on the current block prediction mode, for example, depending on the following conditions: a) whether the current block is an intra-frame or inter-frame coded block; b) the skip_txfm value of at least one neighboring block (e.g., the above and / or left neighboring blocks).

[0173] In some of the above example embodiments, if the current block (or neighboring block) is in intra block copy mode, then when the skip_txfm flag is signaled, the current block (or neighboring block) is regarded as an inter-coded block rather than an intra-predicted block.

[0174] In some of the above example embodiments, a flag referred to as an is_inter flag may be used to indicate whether the current block is an inter-coded block. Such a flag may be signaled prior to the skip_txfm flag in the bitstream so that the decoder may first parse its value and use it to determine the decoding context for the skip_txfm flag that follows in the bitstream.

[0175] In some example embodiments, if the current block is an intra-coded block, one set of contexts may be used to signal the skip_txfm flag. Otherwise, another set of contexts may be used to signal the skip_txfm flag. In other words, two sets of contexts may be designed, and the set of contexts used for signaling may be first selected by the prediction mode of the current block, and then each set of contexts may be further selected based on the other information described above (e.g., adjacent skip_txfm flags and adjacent prediction modes).

[0176] In the above-described embodiments, the context for signaling the skip_txfm flag of the current block and encoding it may depend on its prediction mode and / or the prediction mode of its neighboring blocks, and / or the skip_txfm flag of its neighboring blocks. In some other example embodiments, the signaled is_inter flag and encoding it may depend on the value of the skip_txfm flag of the current block, wherein the is_inter flag is used to indicate whether the current block is an inter-coded block.

[0177] For example, if the skip_txfm flag of the current block is equal to 0 (indicating that at least one coefficient in the current block is non-zero), then one set of contexts can be used to signal the is_inter flag. Otherwise, another set of contexts can be used to signal the is_inter flag. This can be based on the following observation: statistically, blocks with non-zero coefficients can generally have similar prediction modes, and similarly, blocks with all-zero coefficients can also generally have similar prediction modes.

[0178] For another further example, the context for signaling the is_inter flag may alternatively or additionally depend on the neighboring skip_txfm. For example, such context may depend on the following example conditions: a) whether the value of the skip_txfm flag of the current block is zero; b) whether the value of the skip_txfm flag of at least one neighboring block (e.g., the above / left block) is zero. This scheme essentially exploits the statistical correlation between the prediction mode and the coefficient characteristics (all zero or other).

[0179] For these embodiments, due to the interdependencies described above, the skip_txfm flag may be signaled earlier than the is_inter flag in the codestream.

[0180] In some further embodiments, the is_inter flag may not be included in the codestream, and the is_inter flag may be implied to be true (or 1) at the decoder side when the skip_txfm flag is equal to 1. This may be useful because even if there are coefficients that are not transmitted in the codestream due to the skip_txfm flag being 1 (all zero coefficients), inter prediction will be implied or attributed to the block and used for decoding its neighboring blocks when those neighboring blocks are associated with flags encoded based on the prediction mode of the current block.

[0181] Fig.16 A flowchart 1600 of an example method is shown, and the example method follows the basic principles of the above-mentioned implementation method for zero skip flag encoding and decoding. The example method flow starts at 1601. In S1610, the zero skip flag of at least one neighboring block of the current block is determined as a reference zero skip flag. In S1620, the prediction mode for the current block is determined to be an intra-frame prediction mode or an inter-frame prediction mode. In S1630, based on the reference zero skip flag and the prediction mode of the current block, at least one context for decoding the zero skip flag of the current block is derived. In S1640, based on the at least one context, the zero skip flag of the current block is decoded. The example method flow ends at S1699. The above-mentioned method flow is also applied to encoding.

[0182] The embodiments of the present disclosure may be used alone or in any order. In addition, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium. The embodiments of the present disclosure may be applied to luminance blocks or chrominance blocks.

[0183] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.17 A computer system (1700) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0184] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked, or similar mechanisms to create code comprising instructions that may be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through interpretation, microcode execution, etc.

[0185] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, etc.

[0186] Fig.17 The components shown for the computer system (1700) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement on any one component or combination of components illustrated in the exemplary embodiment of the computer system (1700).

[0187] The computer system (1700) may include certain human-machine interface input devices. Such human-machine interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, tapping), visual input (such as gestures), olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still image camera), and videos (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0188] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (1701), mouse (1702), touchpad (1703), touch screen (1710), data gloves (not shown), joystick (1705), microphone (1706), scanner (1707), camera (1708).

[0189] The computer system (1700) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen (1710), a data glove (not shown), or a joystick (1705), but there may also be tactile feedback devices that are not used as input devices), audio output devices (such as: speakers (1709), headphones (not depicted)), visual output devices (such as screens (1710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted), holographic displays, and smoke canisters (not depicted)), and printers (not depicted).

[0190] The computer system (1700) may also include human-accessible storage devices and their associated media, such as optical media including media (1721) such as CD / DVD ROM / RW (1720) with CD / DVD, thumb drives (1722), removable hard drives or solid-state drives (1723), traditional magnetic media such as tapes and floppy disks (not depicted), dedicated ROM / ASIC / PLD-based devices such as security dongles (not depicted), and the like.

[0191] Those skilled in the art will also appreciate that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0192] The computer system (1700) may also include an interface (1754) to one or more communication networks (1755). The network may be, for example, wireless, wired, optical. The network may also be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial networks including CAN buses, and the like. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (1749) (such as, for example, a USB port of the computer system (1700)); other networks are typically integrated into the core of the computer system (1700) by attaching to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1700) can communicate with other entities. Such communications may be one-way receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way, such as to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks may be used on each of those networks and network interfaces as described above.

[0193] The above-mentioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1740) of the computer system (1700).

[0194] The core (1740) may include one or more central processing units (CPUs) (1741), graphics processing units (GPUs) (1742), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (1743), hardware accelerators (1744) for certain tasks, graphics adapters (1750), etc. These devices, along with read-only memory (ROM) (1745), random access memory (1746), internal mass storage (1747) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (1748). In some computer systems, the system bus (1748) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (1748) directly or via a peripheral bus (1749). In one example, a screen (1710) may be connected to a graphics adapter (1750). The architecture of the peripheral bus includes PCI, USB, etc.

[0195] The CPU (1741), GPU (1742), FPGA (1743) and accelerator (1744) can execute certain instructions, and the combination of these instructions can constitute the above-mentioned computer code. The computer code can be stored in ROM (1745) or RAM (1746). Transition data can also be stored in RAM (1746), while permanent data can be stored in, for example, internal mass storage (1747). Fast storage and retrieval of any memory device can be enabled by using cache memory, which can be closely associated with one or more CPUs (1741), GPUs (1742), mass storage (1747), ROM (1745), RAM (1746), etc.

[0196] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0197] As a non-limiting example, a computer system (1700) having an architecture, and in particular a core (1740), can provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage as described above, as well as certain memories of the core (1740) having non-volatile properties, such as a core internal mass storage (1747) or ROM (1745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1740). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can cause the core (1740) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (1746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hard-wiring logic or otherwise embodied in circuitry (e.g., accelerator (1744)) that may replace software or operate in conjunction with software to perform specific processes or specific portions of specific processes described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) storing software for execution, circuitry embodying logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0198] Although the present disclosure describes several exemplary embodiments, there are changes, arrangements, and various alternative equivalents that are within the scope of the present disclosure. In the above-described embodiments and examples, any operation of the process can be combined or arranged in any number or order as needed. In addition, two or more operations of the above-described process can be performed in parallel. Therefore, it is understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0199] Appendix A: Acronyms

[0200] JEM: joint exploration model

[0201] VVC: versatile video coding

[0202] BMS: Benchmark set

[0203] MV: Motion Vector

[0204] HEVC: High Efficiency Video Coding

[0205] SEI: Supplementary Enhancement Information

[0206] VUI: Video Usability Information

[0207] GOPs: Groups of Pictures

[0208] TUs: Transform Units

[0209] PUs: Prediction Units

[0210] CTUs: Coding Tree Units

[0211] CTBs: Coding Tree Blocks

[0212] PBs: Prediction Blocks

[0213] HRD: Hypothetical Reference Decoder

[0214] SNR: Signal Noise Ratio

[0215] CPUs: Central Processing Units

[0216] GPUs: Graphics Processing Units

[0217] CRT: Cathode Ray Tube

[0218] LCD: Liquid-Crystal Display

[0219] OLED: Organic Light-Emitting Diode

[0220] CD: Compact Disc

[0221] DVD: Digital Video Disc

[0222] ROM: Read-Only Memory

[0223] RAM: Random Access Memory

[0224] ASIC: Application-Specific Integrated Circuit

[0225] PLD: Programmable Logic Device

[0226] LAN: Local Area Network

[0227] GSM: Global System for Mobile communications

[0228] LTE: Long-Term Evolution

[0229] CANBus: Controller Area Network Bus

[0230] USB: Universal Serial Bus

[0231] PCI: Peripheral Component Interconnect

[0232] FPGA: Field Programmable Gate Areas

[0233] SSD: solid-state drive

[0234] IC:Integrated Circuit

[0235] HDR: high dynamic range

[0236] SDR: standard dynamic range

[0237] JVET: Joint Video Exploration Team

[0238] MPM: most probable mode

[0239] WAIP: Wide-Angle Intra Prediction

[0240] CU: Coding Unit

[0241] PU: Prediction Unit

[0242] TU: Transform Unit

[0243] CTU: Coding Tree Unit

[0244] PDPC: Position Dependent Prediction Combination

[0245] ISP: Intra Sub-Partitions

[0246] SPS: Sequence Parameter Set

[0247] PPS: Picture Parameter Set

[0248] APS: Adaptation Parameter Set

[0249] VPS: Video Parameter Set

[0250] DPS: Decoding Parameter Set

[0251] ALF: Adaptive Loop Filter

[0252] SAO: Sample Adaptive Offset

[0253] CC-ALF: Cross-Component Adaptive Loop Filter

[0254] CDEF: Constrained Directional Enhancement Filter

[0255] CCSO: Cross-Component Sample Offset

[0256] LSO: Local Sample Offset

[0257] LR: Loop Restoration Filter

[0258] AV1: Alliance for Open Media Video 1 (AOMedia Video 1)

[0259] AV2: Alliance for Open Media Video 2 (AOMedia Video 2)

Claims

1. A video decoding method, characterized in that: The method comprises: Determine a value of a first reference zero skip flag of a first neighboring block of a current block; determining a prediction mode for the current block as an intra prediction mode or an inter prediction mode; deriving at least one first context for decoding a zero skip flag of the current block based on a value of the first reference zero skip flag and a prediction mode of the current block; and Decoding a zero skip flag of the current block according to the at least one first context; Based on the value of the zero skip flag, deriving at least one second context for decoding a prediction mode is_inter flag of the current block; Decoding a prediction mode is_inter flag of the current block according to the at least one second context; and The current block is decoded according to the zero skip flag and the prediction mode is_inter flag.

2. The method according to claim 1, characterized in that Further including: Determine a value of a second reference zero skip flag of a second neighboring block of the current block; Therein, at least one first context for decoding the zero skip flag of the current block is further derived based on the value of the second reference zero skip flag.

3. The method according to claim 2, characterized in that The deriving, based on the value of the first reference zero skip flag and the prediction mode of the current block, at least one first context for decoding the zero skip flag of the current block comprises: in response to the prediction mode of the current block being an inter-frame prediction mode: deriving a first mode-dependent reference zero skip flag based on the first prediction mode of the first neighboring block and the first reference zero skip flag; deriving a second mode-dependent reference zero skip flag based on the second prediction mode of the second neighboring block and the second reference zero skip flag; and The at least one first context is derived based on the first mode-dependent reference zero skipping flag and the second mode-dependent reference zero skipping flag.

4. The method according to claim 3, characterized in that: The at least one first context is derived based on the first mode-related reference zero skip flag and the second mode-related reference zero skip flag, comprising: deriving the at least one first context based on the sum of the first mode-related reference zero skip flag and the second mode-related reference zero skip flag.

5. The method according to claim 2, characterized in that: The first neighboring block and the second neighboring block include a top neighboring block and a left neighboring block of the current block, respectively.

6. The method according to claim 1, characterized in that At least one second context for decoding the prediction mode is_inter flag is further derived based on a first reference zero skip flag of the first neighboring block.

7. The method according to claim 6, characterized in that The first adjacent block includes a top adjacent block or a left adjacent block.

8. The method according to claim 1, characterized in that The deriving, based on the value of the zero skip flag, at least one second context for decoding the prediction mode is_inter flag of the current block comprises: When the value of the zero skip flag is 0, a first set of one or more contexts is selected as the at least one second context; when the value of the zero skip flag is not 0, a second set of one or more contexts is selected as the at least one second context.

9. The method according to claim 1, characterized in that: In the video code stream, the zero skip flag is signaled before the prediction mode is_inter flag.

10. A video encoding method, characterized in that: The method comprises: Determine a value of a first reference zero skip flag of a first neighboring block of a current block; determining a prediction mode for the current block as an intra prediction mode or an inter prediction mode; deriving at least one first context for determining a zero skip flag of the current block based on a value of the first reference zero skip flag and a prediction mode of the current block; and Determining a zero skip flag of the current block according to the at least one first context; Based on the value of the zero skip flag, deriving at least one second context for determining a prediction mode is_inter flag of the current block; Determining a prediction mode is_inter flag of the current block according to the at least one second context; and The current block is encoded according to the zero skip flag and the prediction mode is_inter flag.

11. An electronic device, characterized in that: include: Memory, for storing computer instructions; A processor, configured to call the computer instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A non-volatile computer-readable storage medium for storing instructions, characterized in that: When the instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A method for sending or storing a video stream, characterized in that: The video code stream is generated by encoding according to the method of claim 9, or the video code stream is decoded according to the method of any one of claims 1-8.

Images