Apparatus, method and computer program for video encoding and decoding

Abstract

A method is disclosed, comprising: obtaining a block of transform coefficients; searching for a position of a last non-zero coefficient in a scan order in the block; selecting a secondary transform mode based on the position; and encoding a first syntax element indicating the selected secondary transform mode. A primary transform mode is selected based on the secondary transform mode. A second syntax element indicating the selected primary transform mode and a third syntax element defining transform coefficients within the block are encoded. A method is also disclosed, comprising: receiving encoded information related to a block of transform coefficients; decoding an indication of a position of a last non-zero coefficient in a scan order in the block; and using the position to determine whether an indication of the secondary transform mode is present in a bitstream. If an indication of the secondary transform mode is present in the bitstream, then the first syntax element indicating the secondary transform mode is decoded and used to determine whether an indication of the primary transform mode is present in the bitstream. If an indication of the primary transform mode is present in the bitstream, then the second syntax element indicating the primary transform mode is decoded. If the determined transform mode is present, the transform coefficients are decoded and the determined transform mode is applied to select a transform, and the transform is applied to the transform coefficients.

Description

Technical Field

[0001] The present invention relates to an apparatus, method and computer program for video encoding and decoding. Background Art

[0002] In video codecs, still pictures and pictures in video sequences are usually divided into smaller blocks of samples in order to compress these blocks of samples to the desired file size, bit rate or target quality level. Each block in the block is then predicted by a spatial or temporal component, and the residual error between the predicted block and the corresponding block in the original image is encoded and decoded using a residual codec component. For encoding and decoding, images are usually divided into codec units that define common partitions for prediction and residual codec. A codec unit can also be divided into one or more prediction units that contain sample prediction related information for a complete codec unit or different parts of a codec unit. Similarly, a codec unit can be divided into one or more transform units or transform blocks that contain residual (or transform) codec related information for a specific area or component of a codec unit.

[0003] The residual coding means generally include transforming the residual block into the frequency domain, quantizing the resulting transform coefficients and entropy coding the quantized coefficients. As a primary transform, for example, DCT (Discrete Cosine Transform) or its derivatives can be used.

[0004] So-called secondary transforms can also be applied to further improve the coding efficiency. The secondary transform usually obtains part of the output of the primary transform (e.g., DCT) and performs an additional transform on the low-frequency coefficients of the primary transform. In this case, the final set of transform coefficients includes low-frequency coefficients that have been double-transformed and high-frequency coefficients that have been single-transformed. In some implementations, the high-frequency coefficients can be omitted, and when the secondary transform is enabled for the transform block, only the double transform coefficients can be signaled. As a secondary transform, for example, LFNST (Low Frequency Non-Separable Transform) or its derivatives can be used.

[0005] The selection and signaling of primary transforms typically occurs per transform unit or transform block, while the selection and signaling of secondary transforms typically occurs for a complete codec unit. The availability of secondary transforms for a codec unit is also typically based on the transform coefficients within the entire codec unit reaching a certain frequency level to avoid the often unnecessary signaling of secondary transforms for low activity codec units. This approach imposes restrictions and dependencies between different transform blocks in a codec unit and can lead to increased encoder and decoder side latency and suboptimal codec efficiency. Summary of the invention

[0006] Now, in order to at least alleviate the above problems, an enhanced method for indicating and determining transform modes in encoders and decoders is introduced herein.

[0007] A transform signaling mechanism is provided in which signaling of a primary transform may be performed based on the use of a secondary transform. The signaling may be performed after the last transform coefficient position is signaled in a transform block and before the signaling of coefficients in the transform block. This arrangement may allow a video decoder to determine both the type of primary transform and the type of secondary transform before decoding coefficients in a transform block, and to skip decoding of coefficient regions that are not applicable to a selected primary transform or a selected primary-secondary transform pair.

[0008] The apparatus according to the first aspect comprises at least one processor and at least one memory, the at least one memory having stored thereon codes which, when executed by the at least one processor, cause the apparatus to at least perform:

[0009] obtaining a block of transform coefficients;

[0010] Searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order;

[0011] selecting a secondary transform mode based on the position of the last non-zero coefficient and encoding a first syntax element indicating the selected secondary transform mode;

[0012] selecting a primary transform mode based on the selected secondary transform mode and encoding a second syntax element indicating the selected primary transform mode; and

[0013] The third syntax element defining the transform coefficients within a block is encoded.

[0014] The method according to the second aspect comprises:

[0015] obtaining a block of transform coefficients;

[0016] Searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order;

[0017] selecting a secondary transform mode based on the position of the last non-zero coefficient and encoding a first syntax element indicating the selected secondary transform mode;

[0018] selecting a primary transform mode based on the selected secondary transform mode and encoding a second syntax element indicating the selected primary transform mode; and

[0019] The third syntax element defining the transform coefficients within a block is encoded.

[0020] The apparatus according to the third aspect comprises at least one processor and at least one memory having stored thereon codes which, when executed by the at least one processor, cause the apparatus to at least perform:

[0021] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0022] an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream, in scanning order;

[0023] using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream;

[0024] If an indication of the secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode;

[0025] using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream;

[0026] If an indication of the primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode;

[0027] decoding the transform coefficients; and

[0028] If the determined transform mode exists, the determined transform mode is applied to select the transform, and the transform is applied to the transform coefficients.

[0029] The method according to the fourth aspect comprises:

[0030] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0031] an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream, in scanning order;

[0032] using the search results to determine whether coefficients of a block encoded in the bitstream have been transformed by a secondary transform, and if coefficients of the block encoded in the bitstream have been transformed by a secondary transform, determining a secondary transform mode;

[0033] searching the bitstream for an indication of a primary transform mode;

[0034] using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream;

[0035] If an indication of the secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode;

[0036] using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream;

[0037] If an indication of the primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode;

[0038] decoding the transform coefficients; and

[0039] If the determined transform mode exists, the determined transform mode is applied to select the transform, and the transform is applied to the transform coefficients.

[0040] The apparatus according to the fifth aspect comprises means for:

[0041] obtaining a block of transform coefficients;

[0042] Searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order;

[0043] selecting a secondary transform mode based on the position of the last non-zero coefficient and encoding a first syntax element indicating the selected secondary transform mode;

[0044] selecting a primary transform mode based on the selected secondary transform mode and encoding a second syntax element indicating the selected primary transform mode; and

[0045] The third syntax element defining the transform coefficients within a block is encoded.

[0046] The apparatus according to the sixth aspect comprises means for:

[0047] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0048] an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream, in scanning order;

[0049] using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream;

[0050] If an indication of the secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode;

[0051] using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream;

[0052] If an indication of the primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode;

[0053] decoding the transform coefficients; and

[0054] If the determined transform mode exists, the determined transform mode is applied to select the transform, and the transform is applied to the transform coefficients.

[0055] The computer-readable storage medium according to the seventh aspect comprises code for use by an apparatus, which when executed by a processor causes the apparatus to perform:

[0056] obtaining a block of transform coefficients;

[0057] Searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order;

[0058] selecting a secondary transform mode based on the position of the last non-zero coefficient and encoding a first syntax element indicating the selected secondary transform mode;

[0059] selecting a primary transform mode based on the selected secondary transform mode and encoding a second syntax element indicating the selected primary transform mode; and

[0060] The third syntax element defining the transform coefficients within a block is encoded.

[0061] The computer readable storage medium according to the eighth aspect comprises code for use by an apparatus, which when executed by a processor causes the apparatus to perform:

[0062] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0063] an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream, in scanning order;

[0064] using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream;

[0065] If an indication of the secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode;

[0066] using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream;

[0067] If an indication of the primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode;

[0068] decoding the transform coefficients; and

[0069] If the determined transform mode exists, the determined transform mode is applied to select the transform, and the transform is applied to the transform coefficients.

[0070] The method according to the ninth aspect comprises:

[0071] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0072] determining whether the block of transform coefficients is a first block of transform coefficients in the encoding and decoding unit;

[0073] In response to the block of transform coefficients being a first block of transform coefficients in the codec unit, decoding a secondary transform mode from the bitstream;

[0074] The determined secondary transform mode is applied to select a transform for blocks of all transform coefficients in the codec unit.

[0075] The method according to the tenth aspect comprises:

[0076] receiving a bitstream comprising encoded information associated with a block of transform coefficients;

[0077] determining whether the block of transform coefficients is the first block in the codec unit having one or more non-zero transform coefficients;

[0078] In response to the block of transform coefficients being a first block in the codec unit having one or more non-zero transform coefficients, decoding a secondary transform mode from the bitstream;

[0079] The determined secondary transform mode is applied to select a transform for all blocks in the codec unit having one or more non-zero transform coefficients.

[0080] As described above, the apparatus and the computer-readable storage medium having code stored thereon are thus arranged to perform one or more of the above methods and embodiments related thereto. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] For a better understanding of the present invention, reference will now be made by way of example to the accompanying drawings, in which:

[0082] Figure 1 An electronic device using an embodiment of the present invention is schematically shown;

[0083] Figure 2 Schematically illustrates user equipment suitable for employing embodiments of the present invention;

[0084] Figure 3Also schematically illustrated are electronic devices employing embodiments of the present invention connected using wireless and wired network connections;

[0085] Figure 4 Schematically illustrates an encoder suitable for implementing an embodiment of the present invention;

[0086] Figure 5 illustrates examples of encoder and decoder structures that can be used to encode and decode video and images using primary and secondary transforms according to an embodiment;

[0087] Figure 6 The diagram illustrates a dual transform process in both encoding and decoding directions;

[0088] Figure 7 A double transformation process with an incomplete or partial secondary transformation is shown;

[0089] Figure 8 A flowchart of an encoding method according to an embodiment of the present invention is shown;

[0090] Fig. 9 A flowchart of a decoding method according to an embodiment of the present invention is shown;

[0091] Fig.10 An example of a transform block of size 8×8 and an exemplary scanning order are illustrated;

[0092] Fig.11 A schematic diagram showing a decoder suitable for implementing an embodiment of the present invention; and

[0093] Fig.12 A schematic diagram of an example multimedia communication system is shown in which various embodiments may be implemented. DETAILED DESCRIPTION

[0094] Suitable devices and possible mechanisms for double conversion and inverse double conversion are described in more detail below. In this regard, reference is first made to Figure 1 and Figure 2 ,in Figure 1 A block diagram of a video codec system according to an example embodiment is shown as a schematic block diagram of an example apparatus or electronic device 50 which may include a codec according to an embodiment of the present invention. Figure 2 The layout of the device according to the exemplary embodiment is shown. Figure 1 and Figure 2 components.

[0095] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be appreciated that embodiments of the present invention may be implemented in any electronic device or apparatus that may require encoding and decoding or encoding or decoding video images.

[0096] The device 50 may include a housing 30 for containing and protecting the device. The device 50 may also include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may also include a keyboard 34. In other embodiments of the present invention, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch sensitive display.

[0097] The device may include a microphone 36 or any suitable audio input that can be input as a digital or analog signal. The device 50 may also include an audio output device, which in embodiments of the present invention may be any of the following: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other embodiments of the present invention, the device may be powered by any suitable mobile energy device (such as a solar cell, a fuel cell, or a clockwork generator)). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as (for example) a Bluetooth wireless connection or a USB / Firewire wired connection.

[0098] The apparatus 50 may include a controller 56, a processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which, in embodiments of the invention, may store both data in the form of images and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54 which is suitable for performing or assisting in the encoding and decoding of audio and / or video data.

[0099] The apparatus 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and adapted to provide authentication information for authentication and authorization of the user at the network.

[0100] The device 50 may include a radio interface circuit system 52 connected to the controller and suitable for generating wireless communication signals, for example for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may also include an antenna 44 connected to the radio interface circuit system 52 for transmitting radio frequency signals generated at the radio interface circuit system 52 to (multiple) other devices and for receiving radio frequency signals from (multiple) other devices.

[0101] The apparatus 50 may include a camera capable of recording or detecting individual frames, which are then passed to the codec 54 or controller for processing. The apparatus may receive video image data from another device for processing before transmission and / or storage. The apparatus 50 may also receive images wirelessly or via a wired connection for encoding / decoding. The structural elements of the apparatus 50 described above represent examples of components for performing corresponding functions.

[0102] Relative to Figure 3 , showing an example of a system that may utilize embodiments of the present invention. System 10 includes a plurality of communication devices capable of communicating over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) such as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0103] System 10 may include both wired and wireless communication devices and / or apparatuses 50 suitable for implementing embodiments of the present invention.

[0104] For example, Figure 3 The system shown in shows a representation of a mobile phone network 11 and the Internet 28. Connectivity to the Internet 28 may include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0105] The example communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatuses 50, a combination of a personal digital assistant (PDA) and mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. Apparatus 50 may be stationary or mobile when carried by an individual who is moving. Apparatus 50 may also be located in a mode of transportation, including, but not limited to, a car, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or any similar suitable mode of transportation.

[0106] Embodiments may also be implemented in set-top boxes (i.e., digital TV receivers) that may or may not have display or wireless capabilities; in tablet or (laptop) personal computers (PCs) having hardware or software or a combination of encoder / decoder implementations; in various operating systems; and in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software based encoding and decoding.

[0107] Some or other devices may send and receive calls and messages and communicate with service providers through a wireless connection 25 with a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile phone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0108] The communication device may communicate using various transmission technologies, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-Internet protocol (TCP-IP), short message service (SMS), multimedia message service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. The communication device involved in implementing various embodiments of the present invention may communicate using various media, including but not limited to radio, infrared, laser, cable connection and any appropriate connection.

[0109] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium (such as a wire), while a logical channel can refer to a logical connection through a multiplexed medium capable of transmitting several logical channels. A channel can be used to transmit an information signal (e.g., a bit stream) from one or several senders (or transmitters) to one or several receivers.

[0110] The MPEG-2 Transport Stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video and other media, as well as program metadata or other metadata in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (aka packetized elementary stream) within a TS. Thus, a logical channel within an MPEG-2 TS can be considered to correspond to a specific PID value.

[0111] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and a file format of NAL unit structured video derived from ISOBMFF (ISO / IEC 14496-15).

[0112] A video codec includes an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. The video encoder and / or video decoder may also be separate from each other, i.e., they do not need to form a codec. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0113] A typical hybrid video encoder (e.g., many encoder implementations of ITU-T H.263 and H.264) encodes video information in two stages. First, the pixel values ​​in a particular picture area (or block) are predicted, for example, by a motion compensation component (looking up in one of the previously coded video frames and indicating an area that closely corresponds to the block being coded) or a spatial component (using the pixel values ​​around the block to be coded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is coded. This is typically accomplished by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder is able to control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bit rate).

[0114] In temporal prediction, the prediction source is a previously decoded picture (aka reference picture). In intra-block copy (IBC; aka intra-block copy prediction), prediction is applied similarly to temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer only to temporal prediction, while in other cases, inter-frame prediction may collectively refer to temporal prediction and any of intra-block copy, inter-layer prediction, and inter-view prediction, as long as they are performed in the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0115] Motion compensation can be performed with full sample or sub-sample accuracy. In the case of full sample accurate motion compensation, motion can be represented as motion vectors with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses these displacements to effectively copy samples from a reference picture. In the case of sub-sample accurate motion compensation, the motion vector is represented by fractional or decimal values ​​of the horizontal and vertical components of the motion vector. In the case where the motion vector refers to a non-integer position in the reference picture, a sub-sample interpolation process is usually called to calculate the predicted sample value based on the reference sample and the selected sub-sample position. The sub-sample interpolation process typically includes: horizontal filtering to compensate for the horizontal offset relative to the full sample position, followed by vertical filtering to compensate for the vertical offset relative to the full sample position. However, in some environments, vertical processing can also be performed before horizontal processing.

[0116] Inter prediction (which may be referred to as temporal prediction, motion compensation or motion compensated prediction) reduces temporal redundancy. In inter prediction, the prediction source is a previously decoded picture. Intra prediction exploits the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is typically used for intra coding, in which case inter prediction is not applied.

[0117] One result of the encoding and decoding process is a set of encoding and decoding parameters, such as motion vectors and quantized transform coefficients. If many parameters are first predicted from spatially or temporally adjacent parameters, then these parameters can be entropy encoded and decoded more efficiently. For example, a motion vector can be predicted from spatially adjacent motion vectors, and only the difference relative to the motion vector predictor can be encoded and decoded. Prediction of coding parameters and intra-frame prediction can be collectively referred to as in-picture prediction.

[0118] Figure 4 A block diagram of a video encoder suitable for employing embodiments of the present invention is shown. Figure 4 An encoder for two layers is presented, but it should be appreciated that the presented encoder can be similarly extended to encode more than two layers. Figure 4 An embodiment of a video encoder is illustrated that includes a first encoder section 500 for a base layer and a second encoder section 502 for an enhancement layer. Each of the first encoder section 500 and the second encoder section 502 may include similar elements for encoding an incoming picture. The encoder sections 500, 502 may include a pixel predictor 302, 402, a prediction error encoder 303, 403, and a prediction error decoder 304, 404. Figure 4Embodiments of the pixel predictors 302, 402 are also shown as including inter-frame predictors 306, 406, intra-frame predictors 308, 408, mode selectors 310, 410, filters 316, 416, and reference frame memories 318, 418. The pixel predictor 302 of the first encoder section 500 receives 300 base layer images of the video stream to be encoded at both the inter-frame predictor 306 (which determines the difference between the image and the motion compensated reference frame 318) and the intra-frame predictor 308 (which determines the prediction of the image block based only on the already processed portion of the current frame or picture). The outputs of both the inter-frame predictor and the intra-frame predictor are passed to the mode selector 310. The intra-frame predictor 308 can have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide a prediction signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer picture 300. Correspondingly, the pixel predictor 402 of the second encoder section 502 receives 400 enhancement layer images of the video stream to be encoded at both the inter-frame predictor 406 (which determines the difference between the image and the motion compensated reference frame 418) and the intra-frame predictor 408 (which determines the prediction of the image block based only on the already processed portion of the current frame or picture). The outputs of both the inter-frame predictor and the intra-frame predictor are passed to the mode selector 410. The intra-frame predictor 408 can have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide a prediction signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer picture 400.

[0119] Depending on which coding mode is selected to encode the current block, the output of the inter-frame predictor 306, 406 or the output of one of the optional intra-frame predictor modes or the output of the surface encoder in the mode selector is passed to the output of the mode selector 310, 410. The output of the mode selector is passed to the first summing device 321, 421. The first summing device can subtract the output of the pixel predictor 302, 402 from the base layer picture 300 / enhancement layer picture 400 to generate a first prediction error signal 320, 420, which is input to the prediction error encoder 303, 403.

[0120] The pixel predictor 302, 402 also receives a combination of a predicted representation of the image block 312, 412 and an output 338, 438 of the prediction error decoder 304, 404 from the preliminary reconstructor 339, 439. The preliminary reconstructed image 314, 414 may be passed to the intra predictor 308, 408 and the filter 316, 416. The filter 316, 416 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image 340, 440 which may be stored in a reference frame memory 318, 418. The reference frame memory 318 may be connected to the inter predictor 306 to be used as a reference image with which the future base layer picture 300 is compared in an inter prediction operation. According to some embodiments, subject to the base layer being selected and indicated as a source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer, the reference frame memory 318 may also be connected to the inter-frame predictor 406 to be used as a reference image, with which the future enhancement layer picture 400 is compared in the inter-frame prediction operation. Moreover, the reference frame memory 418 may be connected to the inter-frame predictor 406 to be used as a reference image, with which the future enhancement layer picture 400 is compared in the inter-frame prediction operation.

[0121] According to some embodiments, the filter parameters of the filter 316 from the first encoder section 500 may be provided to the second encoder section 502 subject to the base layer being selected and indicated as the source of filter parameters for predicting the enhancement layer.

[0122] The prediction error encoder 303, 403 comprises a transform unit 342, 442 and a quantizer 344, 444. The transform unit 342, 442 transforms the first prediction error signal 320, 420 into a transform domain. The transform is, for example, a DCT transform. The quantizer 344, 444 quantizes the transform domain signal (eg, DCT coefficients) to form quantized coefficients.

[0123] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs the inverse processing of the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438, which when combined with the predicted representation of the image block 312, 412 at the second summing device 339, 439 produces a preliminary reconstructed image 314, 414. The prediction error decoder can be seen as comprising a dequantizer 361, 461, which dequantizes the quantized coefficient values ​​(e.g. DCT coefficients) to reconstruct the transformed signal, and an inverse transform unit 363, 463, which inversely transforms the reconstructed transformed signal, wherein the output of the inverse transform unit 363, 463 contains (multiple) reconstructed blocks. The prediction error decoder may also include a block filter, which may filter the (multiple) reconstructed blocks according to further decoded information and filter parameters.

[0124] The entropy encoder 330, 430 receives the output of the prediction error encoder 303, 403 and may perform appropriate entropy encoding / variable length encoding on the signal to provide error detection and correction capabilities. The output of the entropy encoder 330, 430 may be inserted into the bitstream, for example, by a multiplexer 508.

[0125] Entropy coding / decoding can be performed in many ways. For example, context-based coding / decoding can be applied, where both the encoder and the decoder modify the context state of the coding / decoding parameters based on the previous coding / decoded coding / decoding parameters. Context-based coding / decoding can be, for example, context-adaptive binary arithmetic coding / decoding (CABAC) or context-based variable length coding / decoding (CAVLC) or any similar entropy coding / decoding. Entropy coding / decoding can be performed alternatively or additionally using a variable length coding / decoding scheme (such as Huffman coding / decoding or exponential Golomb coding / decoding). The decoding of the coding / decoding parameters from the entropy coding / decoding bitstream or codeword can be referred to as parsing.

[0126] The H.264 / AVC standard was developed by the Joint Video Group (JVT) of the Video Coding Experts Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by two parent standardization organizations, and it is called ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are already multiple versions of the H.264 / AVC standard that integrate new extensions or features into the specification. These extensions include scalable video coding (SVC) and multi-view video coding (MVC).

[0127] Version 1 of the High Efficiency Video Codec (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaboration Team on Video Codecs (JCT-VC) of VCEG and MPEG. The standard is published by two parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 Advanced Video Codec (HEVC). Subsequent versions of H.265 / HEVC include scalable, multi-view, range of fidelity, three-dimensional, and screen content codec extensions, which can be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0128] SHVC, MV-HEVC and 3D-HEVC use a common base specification specified in Annex F of Version 2 of the HEVC standard. The common base includes, for example, high-level syntax and semantics, for example specifying some of the features of the layer characteristics of the bitstream, such as inter-layer dependencies; and decoding processes (such as reference picture list construction), including inter-layer reference pictures and picture order count derivation for multi-layer bitstreams. Annex F may also be used in potential subsequent multi-layer extensions of HEVC. It should be understood that even though video encoders, video decoders, encoding methods, decoding methods, bitstream structures and / or embodiments may be described below with reference to specific extensions (such as SHVC and / or MV-HEVC), they are generally applicable to any multi-layer extension of HEVC, and even more generally to any multi-layer video coding and decoding scheme.

[0129] Versatile Video Codec (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Group (JVET) (formally ISO / IEC JTC1 SC29 WG11) of the Motion Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) as the successor to HEVC / H.265.

[0130] Some key definitions, bitstream and codec structures and concepts of H.264 / AVC and HEVC are described in this section as examples of video encoders, decoders, encoding methods, decoding methods and bitstream structures, in which embodiments can be implemented. Some key definitions, bitstream and codec structures and concepts of H.264 / AVC are the same as those in HEVC - therefore, they are described jointly below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but are described for one possible basis on which the present invention can be partially or fully implemented.

[0131] Similar to many earlier video codec standards, the bitstream syntax and semantics and decoding process for error-free bitstreams are specified in H.264 / AVC and HEVC. The encoding process is not specified, but the encoder must generate a conforming bitstream. Bitstream and decoder conformance can be verified using a hypothetical reference decoder (HRD). These standards contain codec tools that help deal with transmission errors and losses, but the use of these tools in encoding is optional, and the decoding process for erroneous bitstreams is not yet specified.

[0132] The basic unit for the input of an H.264 / AVC or HEVC encoder and the output of an H.264 / AVC or HEVC decoder, respectively, is a picture. A picture given as input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture.

[0133] The source picture and the decoded picture each include one or more sample arrays, such as one of the following sample array sets:

[0134] - Luminance (Y) only (monochrome).

[0135] - Luma and two chrominances (YCbCr or YCgCo).

[0136] - Green, Blue and Red (GBR, also known as RGB).

[0137] - Arrays representing other unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0138] In H.264 / AVC and HEVC, a picture can be a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame and can be used as encoder input when the source signal is interlaced. The chroma sample array may not be present (so monochrome sampling may be in use), or the chroma sample array may be subsampled compared to the luma sample array. The chroma format can be summarized as follows:

[0139] - In monochrome sampling, there is only one sample array, which can nominally be considered as a brightness array.

[0140] - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0141] - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array.

[0142] - In 4:4:4 sampling, when no separate color planes are in use, each of the two chroma arrays has the same height and width as the luma array.

[0143] In H.264 / AVC and HEVC, sample arrays can be encoded as separate color planes in the bitstream and the separate encoded color planes can be decoded from the bitstream accordingly. When separate color planes are in use, each of these separate color planes is processed separately (by the encoder and / or decoder) as a picture with monochrome sampling.

[0144] Partitioning can be defined as dividing a set into subsets such that each element of the set is in exactly one of the subsets.

[0145] When describing the operation of HEVC encoding and / or decoding, the following terms may be used. For a certain value of N, a codec block may be defined as an N×N block of samples, such that the partitioning of a codec tree block into codec blocks is a partition. For a certain value of N, a codec tree block (CTB) may be defined as an N×N block of samples, such that the partitioning of a component into codec tree blocks is a partition. A codec tree unit (CTU) may be defined as a codec tree block of luma samples, two corresponding codec tree blocks of chroma samples of a picture with three sample arrays, or a codec tree block of samples of a monochrome picture or picture, which is encoded and decoded using three separate color planes and a syntactic structure for encoding and decoding samples. A codec unit (CTU) may be defined as a codec block of luma samples, two corresponding codec blocks of chroma samples of a picture with three sample arrays, or a codec block of samples of a monochrome picture or picture, which is encoded and decoded using three separate color planes and a syntactic structure for encoding and decoding samples. A CU having a maximum allowed size may be named an LCU (largest codec unit) or a codec tree unit (CTU), and a video picture is divided into non-overlapping LCUs.

[0146] A CU includes one or more prediction units (PUs) defining a prediction process for samples within the CU and one or more transform units (TUs) defining a prediction error coding process for samples in the CU. Typically, a CU includes a square block of samples whose size can be selected from a predefined set of possible CU sizes. Each PU and TU can also be divided into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes accordingly. Each PU has prediction information associated with it that defines what kind of prediction to apply for pixels within the PU (e.g., motion vector information for inter-prediction PUs and intra-prediction directionality information for intra-prediction PUs).

[0147] Each TU can be associated with information describing the prediction error decoding process for samples within the TU, including, for example, DCT coefficient information. It is usually signaled at the CU level whether prediction error coding is applied for each CU. In the absence of a prediction error residual associated with a CU, it can be considered that the TU does not exist for the CU. The division of an image into CUs and the division of a CU into PUs and TUs is usually signaled in the bitstream, allowing a decoder to reproduce the expected structure of these units.

[0148] In HEVC, pictures can be divided into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, the division of tiles forms a standard grid, where the height and width of the tiles differ from each other by a maximum of one LCU. In HEVC, a slice is defined as an integer number of codec tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) in the same access unit. In HEVC, a slice segment is defined as an integer number of codec tree units that are ordered consecutively in a tile scan and contained in a single NAL unit. The division of each picture into slice segments is a partition. In HEVC, an independent slice segment is defined as a slice segment for which the values ​​of the syntactic elements of the slice segment header are not inferred from the values ​​of the previous slice segment, and a dependent slice segment is defined as a slice segment for which the values ​​of some syntactic elements of the slice segment header are inferred from the values ​​of the independent slice segment preceding in decoding order. In HEVC, a slice header is defined as a slice segment header of an independent slice segment, which is the current slice segment or an independent slice segment before the current dependent slice segment, and a slice segment header is defined as a part of a codec slice segment containing data elements related to the first or all codec tree units represented in the slice segment. If a tile is not in use, a CU is scanned within a tile or within a picture in the raster scan order of an LCU. Within an LCU, a CU has a specific scanning order.

[0149] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error coding to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it for display and / or storing it as a prediction reference for upcoming frames in the video sequence.

[0150] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF).H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0151] In a typical video codec, motion information is indicated by a motion vector associated with each motion compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement of an image block in a picture to be coded (on the encoder side) or decoded (on the decoder side) and a prediction source block in one of the pictures previously coded or decoded. In order to efficiently represent motion vectors, these motion vectors are usually differentially coded relative to block-specific predicted motion vectors. In a typical video codec, predicted motion vectors are created in a predefined manner, such as calculating the median of the coded or decoded motion vectors of adjacent blocks. Another way to create motion vector predictions is to generate a candidate prediction list from adjacent blocks and / or co-located blocks in a temporal reference picture and signaling the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is also possible to predict which reference picture(s) are used for motion compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously coded / decoded picture. The reference index is usually predicted based on adjacent blocks and / or co-located blocks in a temporal reference picture. Furthermore, typical high efficiency video codecs employ an additional motion information encoding / decoding mechanism, often referred to as merging / merge mode, in which all motion field information (the motion field information includes motion vectors and corresponding reference picture indexes for each available reference picture list) is predicted and used without any modification / correction. Similarly, the predicted motion field information is performed using the motion field information of neighboring blocks and / or co-located blocks in the temporal reference picture, and the used motion field information is signaled in a list of motion field candidate lists, which is populated with the motion field information of the available neighboring / co-located blocks.

[0152] In a typical video codec, the prediction residuals after motion compensation are first transformed with a transform kernel (such as DCT) and then encoded and decoded. The reason for this is that there is usually still some correlation between the residuals, and the transform can help reduce this correlation and provide more efficient encoding and decoding in many cases.

[0153] Typical video encoders use a Lagrangian cost function to find the best codec mode, e.g., the desired codec mode for a block and associated motion vectors. This cost function uses a weighting factor λ to relate the (exact or estimated) image distortion due to the lossy codec to the (exact or estimated) amount of information needed to represent the pixel values ​​in the image region:

[0154] C=D+λR (1)

[0155] where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean square error) taking into account mode and motion vectors, and R is the number of bits required to represent the required data to reconstruct the image block in the decoder (including the amount of data representing the candidate motion vectors).

[0156] Video codec standards and specifications may allow encoders to divide codec pictures into codec slices, etc. In-picture prediction is typically disabled across slice boundaries. Therefore, slices can be viewed as a way to divide codec pictures into independently decodable fragments. In H.264 / AVC and HEVC, in-picture prediction can be disabled across slice boundaries. Therefore, slices can be viewed as a way to divide codec pictures into independently decodable fragments, and slices are typically viewed as basic units of transmission. In many cases, the encoder can indicate in the bitstream which types of in-picture prediction are turned off across slice boundaries, and the decoder operation takes this information into account, for example, when determining which prediction sources are available. For example, if neighboring CUs are located in different slices, samples from neighboring CUs may be considered unavailable for intra prediction.

[0157] The basic unit for the output of H.264 / AVC or HEVC encoders and the input of H.264 / AVC or HEVC decoders, respectively, is the network abstraction layer (NAL) unit. In order to be transmitted over a packet-oriented network or stored in a structured file, the NAL unit can be encapsulated into a packet or similar structure. In H.264 / AVC and HEVC, a byte stream format has been specified for a transmission or storage environment that does not provide a frame structure. The byte stream format separates NAL units from each other by attaching a start code in front of each NAL unit. In order to avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm, which adds emulation prevention bytes to the NAL unit payload if the start code will appear otherwise. In order to achieve direct gateway operation between packet-oriented systems and stream-oriented systems, start code emulation prevention can always be performed regardless of whether the byte stream format is in use. The NAL unit can be defined as a syntactic structure that contains an indication of the type and byte of the data to be followed, which contains the data in the form of an RBSP interspersed with emulation prevention bytes when necessary. A Raw Byte Sequence Payload (RBSP) may be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP is either empty or in the form of a data bit string containing syntax elements, followed by an RBSP stop bit, and followed by zero or more subsequent bits equal to 0.

[0158] The NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0159] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains a reserved bit, a six-bit NAL unit type indication, a three-bit nuh_temporal_id_plus1 indication for the temporal level (which may need to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be regarded as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1–1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest temporal level. The value of temporal_id_plus1 must be non-zero to avoid start code emulation involving two NAL unit header bytes. The bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to the selected value and including all other VCL NAL units is consistent. Therefore, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as inter prediction reference. A sublayer or temporal sublayer can be defined as a temporal scalable layer (or temporal layer, TL) of a temporal scalable bitstream, which consists of VCL NAL units with a specific value of the TemporalId variable and associated non-VCL NAL units. nuh_layer_id can be understood as a scalability layer identifier.

[0160] NAL units can be classified into video codec layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually codec slice NAL units. In HEVC, VCL NAL units contain syntax elements representing one or more CUs.

[0161] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end-of-sequence NAL unit, an end-of-bitstream NAL unit, or a filler data NAL unit. Parameter sets may be required for reconstruction of decoded pictures, while many other non-VCL NAL units are not necessary for reconstruction of decoded sample values.

[0162] Parameters that remain unchanged through the encoded video sequence may be included in a sequence parameter set. In addition to the parameters that may be required for the decoding process, the sequence parameter set may optionally include video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that can be referenced by one or more picture parameter set RBSPs or one or more SEI NAL units containing a buffering period SEI message. The picture parameter set contains such parameters that may not change in several encoded pictures. The picture parameter set RBSP may include parameters that can be referenced by the codec slice NAL units of one or more encoded pictures.

[0163] In HEVC, a video parameter set (VPS) can be defined as a syntactic structure containing syntax elements that apply to zero or more complete codec video sequences, as determined by the content of the syntax elements found in the SPS, which are referenced by syntax elements found in the PPS, which are referenced by syntax elements found in each slice segment header.

[0164] A video parameter set RBSP may include parameters that can be referenced by one or more sequence parameter set RBSPs.

[0165] The relationship and hierarchy between a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS) can be described as follows. The VPS is located at a level above the SPS in the parameter set hierarchy and in the context of scalability and / or 3D video. The VPS may include parameters common to all slices in all (scalability or view) layers in a complete coded video sequence. The SPS includes parameters common to all slices in a specific (scalability or view) layer in a complete coded video sequence, and may be shared by multiple (scalability or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalability or view layer in an access unit), and may be shared by all slices in a multi-layer representation.

[0166] The VPS may provide information about dependencies of layers in the bitstream and many other information applicable to all slices in all (scalability or view) layers in a complete coded video sequence. The VPS may be viewed as comprising two parts, a base VPS and a VPS extension, where the VPS extension may optionally be present.

[0167] Out-of-band transmission, signaling or storage can additionally or alternatively be used for purposes other than tolerance to transmission errors, such as ease of access or session negotiation. For example, sample entries for a track in a file conforming to the ISO base media file format may include parameter sets, while codec data in the bitstream is stored elsewhere in the file or in another file. The phrases along the bitstream (e.g., indicated along the bitstream) or along a codec unit of a bitstream (e.g., indicated along a codec tile) may be used in the claims and described embodiments to refer to out-of-band transmission, signaling or storage in a manner in which the out-of-band data is associated with the bitstream or codec unit, respectively. The phrases decoding along the bitstream or along a codec unit of a bitstream, etc. may refer to decoding the referenced out-of-band data (which may be obtained from the out-of-band transmission, signaling or storage) associated with the bitstream or codec unit, respectively.

[0168] A SEI NAL unit may contain one or more SEI messages that are not necessary for decoding of output pictures but may assist in related processes such as picture output timing, rendering, error detection, error concealment, and resource reservation.

[0169] A codec picture is a codec representation of a picture.

[0170] In HEVC, a codec picture can be defined as a codec representation of a picture that contains all codec tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any specific value of nuh_layer_id. In addition to VCL NAL units containing codec pictures, access units can also contain non-VCL NAL units. The specified classification rule can, for example, associate pictures with the same output time or picture output count value into the same access unit.

[0171] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or a byte stream that forms a representation of a codec picture and associated data (which forms one or more codec video sequences). The first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video codecs) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which can be called an end-of-bitstream (EOB) NAL unit and is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have a nuh_layer_id equal to 0.

[0172] In H.264 / AVC, a codec video sequence is defined as a contiguous sequence of access units in decoding order from an IDR access unit (inclusive) to the next IDR access unit (exclusive) or the end of the bitstream, whichever comes first.

[0173] In HEVC, a Coded Video Sequence (CVS) may be defined, for example, as a sequence of access units consisting, in decoding order, of an IRAP access unit with NoRaslOutputFlag equal to 1, followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1, including up to all subsequent access units, but not including any subsequent access units that are IRAP access units with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit where the base layer picture is an IRAP picture. The value of NoRaslOutputFlag is equal to 1 for each IDR picture, each BLA picture, and each IRAP picture that is the first picture in decoding order in that particular layer in the bitstream, the first IRAP picture after the end-of-sequence NAL unit, in decoding order with the same value of nuh_layer_id. There may be a component that provides the value of HandleCraAsBlaFlag to the decoder from an external entity that may control the decoder, such as a player or receiver. For example, HandleCraAsBlaFlag can be set to 1 by a player that seeks to a new position in the bitstream or tunes to a broadcast and starts decoding, then starts decoding from a CRA picture. When HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is processed and decoded like a BLA picture.

[0174] In HEVC, a codec video sequence may additionally or alternatively (to the above specification) be designated as ending when a specific NAL unit, which may be referred to as an end-of-sequence (EOS) NAL unit, appears in the bitstream and has nuh_layer_id equal to 0.

[0175] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures have been decoded. An open GOP is a group of pictures where, when decoding starts from the initial intra picture of an open GOP, pictures before the initial intra picture in the output order may not be decoded correctly. In other words, pictures of an open GOP may refer to pictures belonging to a previous GOP (in inter prediction). The HEVC decoder is able to identify the intra picture that starts an open GOP because a specific NAL unit type, the CRA NAL unit type, can be used for its codec slice. A closed GOP is a group of pictures where, when decoding starts from the initial intra picture of a closed GOP, all pictures can be decoded correctly. In other words, no picture in a closed GOP refers to any picture in a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start from an IDR picture. In HEVC, a closed GOP can also start from a BLA_W_RADL or BLA_N_LP picture. Due to the greater flexibility in selecting reference pictures, an open GOP codec structure may be more efficient in compression than a closed GOP codec structure.

[0176] The decoded picture buffer (DPB) can be used in the encoder and / or decoder. There are two reasons to buffer the decoded pictures, for reference in inter-frame prediction and for reordering the decoded pictures to output order. Since H.264 / AVC and HEVC provide great flexibility for both reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Therefore, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. When the decoded picture is no longer used as a reference and does not need to be output, the decoded picture can be removed from the DPB.

[0177] In many codec modes of H.264 / AVC and HEVC, reference pictures used for inter prediction are indicated using indices to reference picture lists. The indices can be coded using variable length codecs, which generally results in smaller indices having shorter values ​​for the corresponding syntax elements. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bi-predictive (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.

[0178] Many codec standards (including H.264 / AVC and HEVC) may have a decoding process to obtain a reference picture index of a reference picture list, which may be used to indicate which of multiple reference pictures is used for inter prediction of a particular block. The reference picture index may be encoded into the bitstream by the encoder, or may be obtained (by the encoder and decoder) in some other inter-frame codec modes, for example using neighboring blocks.

[0179] The motion parameter type or motion information may include but is not limited to one or more of the following types:

[0180] - an indication of the prediction type (e.g. intra prediction, uni prediction, bi prediction) and / or the number of reference pictures;

[0181] - indication of prediction direction, such as inter (aka temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP) and inter-component prediction (which may be indicated per reference picture and / or per prediction type, and in some embodiments inter-view and view synthesis prediction may be jointly considered as one prediction direction) and / or

[0182] - an indication of the reference picture type, such as a short-term reference picture and / or a long-term reference picture and / or an inter-layer reference picture (which may be indicated, for example, per reference picture)

[0183] - a reference index to a reference picture list and / or any other identifier of a reference picture (which may be indicated, for example, per reference picture and its type may depend on the prediction direction and / or the reference picture type and may be accompanied by other relevant information fragments, such as the reference picture list to which the reference index applies, etc.);

[0184] - horizontal motion vector component (which may be indicated, for example, per prediction block or per reference index, etc.);

[0185] - vertical motion vector component (which may be indicated, for example, per prediction block or per reference index, etc.);

[0186] - one or more parameters (such as picture order count difference and / or relative camera spacing between a picture containing or associated with motion parameters and its reference pictures) which may be used for scaling of horizontal motion vector components and / or vertical motion vector components in one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference picture or per reference index, etc.);

[0187] - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left sample of the block in units of luma samples;

[0188] - The extent (eg width and height) of the block to which the motion parameters and / or motion information apply.

[0189] Scalable video codec can refer to a codec structure in which a bitstream can contain multiple representations of the content, such as at different bit rates, resolutions or frame rates. In these cases, the receiver can extract the desired representation based on its characteristics (such as the resolution that best matches the display device). Alternatively, a server or network element can extract the portion of the bitstream to be sent to the receiver based on, for example, the network characteristics or processing capabilities of the receiver. A meaningful decoded representation can be generated by decoding only a specific portion of the scalable bitstream. A scalable bitstream is typically composed of a "base layer" that provides the lowest quality video available and one or more enhancement layers that enhance the video quality when received and decoded with the lower layers. In order to improve the codec efficiency of the enhancement layer, the codec representation of the layer is typically dependent on the lower layer. For example, the motion and mode information of the enhancement layer can be predicted from the lower layer. Similarly, the pixel data of the lower layer can be used to create a prediction for the enhancement layer.

[0190] In some scalable video codec schemes, a video signal can be encoded into a base layer and one or more enhancement layers. The enhancement layer can enhance, for example, the temporal resolution (i.e., frame rate), the spatial resolution, or simply improve the quality of the video content represented by another layer or a portion thereof. Each layer together with all its subordinate layers is a representation of the video signal, for example, at a specific spatial resolution, temporal resolution, and quality level. In this document, we refer to the scalable layer together with all its subordinate layers as a "scalable layer representation." Portions of the scalable bitstream corresponding to the scalable layer representation can be extracted and decoded to produce a representation of the original signal at a specific fidelity.

[0191] Scalability modes or scalability dimensions may include, but are not limited to, the following:

[0192] - Quality scalability: Base layer pictures are encoded with lower quality than enhancement layer pictures, which can be achieved, for example, by using larger quantization parameter values ​​(i.e., larger quantization step sizes for transform coefficient quantization) in the base layer than in the enhancement layer. Quality scalability can also be classified as fine granularity or fine granularity scalability (FGS), medium granularity or medium granularity scalability (MGS), and / or coarse granularity or coarse granularity scalability (CGS), as described below.

[0193] - Spatial scalability: Base layer pictures are coded at a lower resolution (ie, have fewer samples) than enhancement layer pictures. Spatial scalability and quality scalability, especially its coarse-grained type, can sometimes be considered the same type of scalability.

[0194] - Bit depth scalability: Base layer pictures are encoded or decoded at a lower bit depth (eg 8 bits) than enhancement layer pictures (eg 10 or 12 bits).

[0195] - Dynamic range scalability: Scalable layers represent different dynamic ranges and / or images obtained using different tone mapping functions and / or different optical transfer functions.

[0196] - Chroma format scalability: Base layer pictures provide lower spatial resolution in the chroma sample array (eg, encoded in 4:2:0 chroma format) compared to enhancement layer pictures (eg, 4:4:4 format).

[0197] - Color gamut scalability: The enhancement layer pictures have a richer / wider color representation range than the base layer pictures - for example the enhancement layer may have the UHDTV (ITU-R BT.2020) color gamut and the base layer may have the ITU-R BT.709 color gamut.

[0198] - View scalability, which may also be referred to as multi-view coding. The base layer represents the first view, while the enhancement layer represents the second view. A view may be defined as a sequence of pictures representing one camera or viewpoint. It may be thought of as in stereo or dual-view video, one video sequence or view is presented to the left eye, while parallel views are presented to the right eye.

[0199] - Depth scalability, which may also be referred to as depth enhancement codec. A layer or some layers of the bitstream may represent texture view(s) while other layer or layers may represent depth view(s).

[0200] - Region of Interest Scalability (as described below)

[0201] - Interlaced to progressive scalability (also known as field to frame scalability): Codecd interlaced source content material of a base layer is enhanced by an enhancement layer to represent progressive source content. The codecd interlaced source content in the base layer may include codec fields, codec frames representing pairs of fields, or a mixture of them. In interlaced to progressive scalability, base layer pictures may be resampled so that they become appropriate reference pictures for one or more enhancement layer pictures.

[0202] - Hybrid codec scalability (also known as codec standard scalability): In hybrid codec scalability, the bitstream syntax, semantics, and decoding process of the base layer and enhancement layer are specified in different video codec standards. Therefore, the base layer pictures are coded or decoded according to a different codec standard or format than the enhancement layer pictures. For example, the base layer can be coded or decoded with H.264 / AVC, while the enhancement layer can be coded or decoded with HEVC multi-layer extension.

[0203] It should be understood that many scalability types can be combined and applied together. For example, color gamut scalability and bit depth scalability can be combined.

[0204] The term layer may be used in the context of any type of scalability, including view scalability and depth enhancement. An enhancement layer may refer to any type of enhancement, such as SNR, spatial, multi-view, depth, bit depth, chroma format, and / or color gamut enhancement. A base layer may refer to any type of base video sequence, such as a base view, a base layer for SNR / spatial scalability, or a texture base view for depth enhanced video codecs.

[0205] As described above, still pictures and pictures in a video sequence are usually divided into smaller blocks of samples, which are predicted by spatial or temporal components, and the residual error between the predicted block and the corresponding block in the original image is encoded and decoded using a residual coding component. The residual coding component usually includes transforming the residual block into the frequency domain (using, for example, DCT (discrete cosine transform) or its derivative as a primary transform), quantizing the resulting transform coefficients, and entropy coding and decoding the quantized coefficients.

[0206] A so-called secondary transform can also be applied to further improve the coding efficiency. A secondary transform typically takes a portion of the output of a primary transform (eg, DCT) and performs an additional transform on the low frequency coefficients of the primary transform. Figure 5 The example of the encoder 530 and decoder 540 structure that can be used to encode and decode video and images using primary and secondary transforms is illustrated. The encoder 530 can implement logic to select the desired transform mode: mts_idx for primary transform and lfnst_idx for secondary transform. These modes are encoded in the bitstream. The decoder 540 can then decode the same transform mode from the bitstream representation and apply the selected secondary transform and primary transform to the decoded transform coefficients. The secondary transform mode lfnst_idx equal to zero can indicate bypassing the secondary transform and only performing primary transform on the transform coefficients. The primary transform can also be bypassed in the video codec. This can be indicated by sending a separate mark for enabling this transform skip mode, or the indication can be embedded in the signaling of the primary transform index mts_idx using one of its values ​​as an indication of the transform bypass mode.

[0207] Reference again Figure 4 , wherein the transform units 342 and 442 can be regarded as comprising Figure 5 The primary transform unit 531 and the secondary transform unit 532 of the dual transform encoder 530. Similarly, Figure 4 The inverse transform units 362 and 462 in the embodiment may be regarded as comprising Figure 5 The inverse secondary transform unit 543 and the inverse primary transform unit 544 of the dual transform decoder 540.

[0208] The input to the dual transform encoder 530 typically includes residual sample values, which define the difference between the predicted sample values ​​and the original sample values ​​in the video or image encoder. These values ​​are passed through two transform processes called primary transform 531 and secondary transform 532. The resulting transform coefficients can be quantized 533 to the desired accuracy, entropy encoded 534 and output to a bitstream 536 or file. The dual transform decoder 540 typically reads the encoded symbols from the bitstream, entropy decodes 541 and dequantizes 542 it to transform coefficients and performs the inversion of the secondary transform 543 and the inversion of the primary transform 544. The primary transform can be, for example, a 2-dimensional DCT or DST, and the secondary transform can be, for example, a non-separated 2-dimensional transform.

[0209] The transform selector 535 may perform primary transform mode selection and secondary transform mode selection and prepare an indication of the selected transform mode to signal to the decoder. The operation of the transform selector 535 according to several embodiments will be described later in this specification.

[0210] Improved methods for the double transform and inverse double transform processes are now introduced.

[0211] Figure 8 An example of the operation of the encoder 530 according to an embodiment is illustrated. The encoding method includes inputting (800) a residual sample block; determining (801) the position of the last non-zero coefficient in scanning order; determining (802) a secondary transform mode based on the position of the last non-zero coefficient; determining (803) a primary transform mode based on the secondary transform mode; determining (804) transform coefficients; and performing (805) a primary transform and a secondary transform.

[0212] Fig. 9An example of the operation of a decoder 540 according to an embodiment is illustrated. The decoding method includes receiving (900) a bitstream and decoding (901) a last_pos syntax element to determine a last coefficient position within a transform block. The determined last coefficient position within the transform block is used to check (902) whether the last position indicates that a lfnst_idx syntax element is signaled in the bitstream. If last_pos is within a determined range, then the lfnst_idx syntax element is decoded (903) and a secondary transform mode is determined based on the decoded lfnst_idx syntax element. If last_pos is not within the determined range, then the lfnst_idx syntax element is set (904) to zero or another predetermined value, which indicates that the lfnst_idx syntax element is not signaled in the bitstream, and the secondary transform mode is determined to be absent. The decoded (903) or set (904) value of the lfnst_idx syntax element is also checked (905) to determine whether the lfnst_idx syntax element indicates mts_idx signaling. If the check reveals that the lfnst_idx syntax element indicates mts_idx signaling, then the mts_idx syntax element is decoded (906) and the primary transform mode is determined based on the secondary transform mode. If lfnst_idx is not within the determined range, then lfnst_idx is set (907) to 0 or another predetermined value indicating that the primary transform mode is determined to be absent. The transform coefficients are then decoded (908) and the selected (existing) primary transform and secondary transform are selected (909).

[0213] In the following, some embodiments related to signaling a primary transform mode and a secondary transform mode in conjunction with a video or image codec, encoder or decoder will be described in more detail.

[0214] The encoder 530 obtains a transform block including residual samples for processing. The transform selector 535 determines (searches) the position of the last coefficient in the transform block. Determining the position of the last transform coefficient can be done in the encoder by scanning the coefficients in a predetermined order and checking what the position of the last non-zero coefficient or another coefficient that meets a specific activity criterion is in the transform block in the scan order. For example, the determination can be performed so that, for example, the sample value of the transform block is checked starting from the last coefficient in the transform block, and if the sample value is zero, then the sample value of the previous coefficient in the reverse scan order is checked to determine whether it is zero. Repeat this situation until the first non-zero coefficient is found. The location of the coefficient in the transform block is used as an indication of the last non-zero coefficient. The location can be indicated, for example, as a coordinate (the row and column of the location of the last coefficient, or as a sequential number of coefficients in the scan order or in some other way). Different variants of diagonal scanning can be used in image and video coding applications, because these variants combined with the signaling of the last non-zero coefficient in the block can efficiently represent the typical coefficient distribution inside the transform block, where the coefficients are usually packed to the beginning of the scan and the upper left corner of the transform block. exist Fig.10 An example of such a scan is given in . In this example, a scan is defined for a block 1200 of 8×8 transform coefficients based on 4×4 coefficient groups, where the scan order for the groups is defined as a diagonal from the top right corner to the bottom left corner, and the scan order within the coefficient groups is also defined in the same general direction. In this example, the numbers in the transform block 1200 illustrate the order in which the samples are scanned, which starts at 0 and continues to 63. In this example, the scan order follows a diagonal scan order from the top right corner to the bottom left corner of the block, as indicated by arrow 1201. Within a block, a similar scan order may be followed, such as from Fig.10 In other words, the coefficients in the block in the upper left corner are first scanned in a diagonal scanning order from the upper right corner to the lower left corner, then the coefficients in the block in the upper right corner are scanned in a diagonal scanning order from the upper right corner to the lower left corner, then the coefficients in the block in the lower left corner are scanned, and finally the block in the lower right corner is scanned. As the name implies, the reverse scanning order mentioned above is the opposite of the scanning order described above. Using Fig.10 For example, the last coefficient is numbered 63, the previous coefficient is numbered 62, and so on.

[0215] Instead of a non-zero value of the coefficient, the activity criterion mentioned above can also be any kind of activity measure. For example, it can be a threshold value, wherein instead of searching for the first non-zero value as explained above, the search can include finding the first coefficient having a value (or an absolute value in case negative values ​​may be present) exceeding the threshold value. The activity measure can also be, for example, the total number of non-zero coefficients or the sum of the absolute values ​​of the coefficients.

[0216] The encoder can encode the syntactic elements describing the coordinates of the position after determining the position of the last non-zero coefficient and output it to the bitstream. The corresponding decoder can then decode these coordinates and decode the non-zero coefficients of the transform block starting from this position in reverse scanning order towards the beginning of the scan. In other words, the decoder can decode the values ​​of the last non-zero coefficient and other coefficients of the transform block before the last non-zero coefficient.

[0217] When the position of the last non-zero coefficient in a transform block has been determined, the position can be used to determine the secondary transform mode. There may be several alternatives for how the position affects the secondary transform mode. Some examples will be provided later in this specification.

[0218] The determined secondary transformation mode can then be used to select the primary transformation mode. There may be several alternatives for how the secondary transformation mode affects the primary transformation mode. Some examples will be provided later in this specification.

[0219] When the secondary transform mode and the first transform mode have been selected, transform coefficients of a transform block can be determined and a transform operation based on the transform coefficients, the primary transform mode, and the secondary transform mode can be performed.

[0220] In a video or image encoder, determining the secondary transform mode can be done, for example, by applying different transform candidates to the prediction residual and selecting the transform candidate that minimizes a given cost function. Such a cost function can contain a term that depends on the exact or estimated number of bits required to represent the transform block, transform unit or coding unit with the candidate; and a term that depends on a measure of the exact or estimated reconstruction error of the sample block.

[0221] In a video or image decoder, determining the secondary transform mode can be done by parsing a syntax element indicating the secondary transform mode.

[0222] Determining the secondary transform mode can include different availability checks in both the encoder and the decoder; and the signaling and parsing of the secondary transform mode can be omitted under some conditions. For example, it can be defined that the last coefficient in the transform block must be within a certain range in scan order to be present in the bitstream with secondary transform signaling. In case the signaling is omitted, the secondary transform mode can be determined, for example, as zero or another indication, which can be used to indicate that the secondary transform is turned off for a transform block, transform unit or codec unit.

[0223] The selection of a primary transform mode in a video or image encoder can be done in a similar way as the selection of a secondary transform mode, including testing different primary transform modes and evaluating a cost function and selecting the mode that minimizes the selected cost function.

[0224] In an embodiment, determining the primary transform mode is done based on the determined secondary transform mode. This may include, for example, omitting the signaling of some or all syntax elements related to the primary transform mode in the case where a non-zero secondary transform mode is included in the bitstream. The signaling of the primary transform mode can also be divided into two or mode parts in the bitstream. For example, a so-called transform skip mode in which the primary transform and the secondary transform are bypassed can be done before signaling the position of the last coefficient in the transform block, while signaling the secondary transform mode can be done after the last coefficient position signaling, followed by signaling the primary transform in the case where the secondary transform is turned off for the transform block.

[0225] The transform coefficients of the transform block can be determined in various ways. These transform coefficients can be signaled in a specific scan order or inverse scan order, for example, using context-based arithmetic coding for quantized coefficient values. Prior to performing an inverse primary transform operation and / or a secondary transform operation, the quantized coefficients signaled in the bitstream can be dequantized and scaled in various ways.

[0226] Performing the transform operation can also be done in different ways. The transform or inverse transform can be performed, for example, using matrix multiplication operations, multiplication and addition operations, or shift and addition operations, or a combination thereof. In addition to parsing the secondary transform mode and the primary transform mode, the transform matrix selection can also include operations and conditions such as checking the dimensions of the transform block, the luminance / chrominance channels of the transform block, or the intra-prediction mode associated with the transform block, transform unit, prediction unit, or coding unit.

[0227] The primary transform can refer to a transform applied to a decoded transform coefficient or a transform set of transform coefficients output from a secondary transform in a decoder. The secondary transform can refer to a transform applied to a decoded transform coefficient and the output is fed as an input to the primary transform. Examples of primary transforms include transforms of the discrete cosine transform (DCT) and discrete sine transform (DST) series, which are used, for example, in the VVC draft standard and signaled using the tu_mts_idx syntax element. The VVC draft standard also includes a secondary transform called a low-frequency non-separated transform or LFNST. Enabling and signaling the LFNST secondary transform mode is accomplished using the lfnst_idx syntax element in the code implemented according to the draft standard.

[0228] In an embodiment, the signaling or decoding of the primary transform is performed based on the use of the secondary transform. The signaling or decoding is advantageously performed after the last transform coefficient position is signaled or decoded in the transform block and before the coefficients in the transform block are signaled or decoded. This arrangement allows the video decoder to determine both the type of the primary transform and the type of the secondary transform before decoding the coefficients in the transform block, and skip decoding of coefficient regions that are not applicable to the selected primary transform or the selected primary-secondary transform pair.

[0229] There may be several benefits when performing both primary transform signaling and secondary transform signaling and decoding at the transform block level and determining the primary transform mode based on knowledge of the selected secondary transform mode. First, signaling the secondary transform mode at the codec unit level after encoding or decoding the entire codec unit may increase the latency achieved. For example, in an encoder, the availability criteria for a secondary transform in the luma component may only be met after the chroma components of the codec unit are also encoded. This may complicate the encoder design and may force the encoder to buffer information that may be irrelevant. Similarly, in a decoder, the luma transform may only be determined after the syntax elements of all chroma transform blocks within the same codec unit have been decoded. Second, in the usage approach, first indicating the secondary transform mode and being able to omit the signaling of the primary transform mode may be more efficient for codec efficiency, especially when the use of the secondary transform is restricted to a subset of the primary transform.

[0230] Different thresholds and ranges of syntax element values ​​can be used to determine whether a particular syntax element is present in a bitstream. In addition, other parameters, such as the use of a particular intra- or inter-prediction mode for a codec unit, prediction unit, transform unit, or block, can be included in the decision to decode or encode a secondary transform or primary transform mode. As an example, the following syntax structure can be used to represent the bitstream syntax:

[0231]

[0232]

[0233] The general conditions for the secondary transform indication lfnstGenericConditions can include, for example, comparing the dimensions of the transform block with a threshold. For example, it can include comparing the minimum of the width and height of the transform block with a value of 4 or 2. In addition to the general conditions, the conditions for parsing the secondary transform mode lfnst_idx can include checking the last coefficient position in the transform block. An example of this situation is also given in the syntax table by defining the lfnstLastPosConditions variable, which is calculated based on the last scan position lastScanPos of the transform block. These conditions, separated or together, can be referred to as secondary transform availability conditions.

[0234] Similarly, the conditions for decoding the primary transform mode mts_idx may include generic conditions mtsGenericConditions, such as the minimum or maximum dimensions of the transform block. In addition to the generic conditions, the indicated lfnst_idx may be included in the availability check of the mts_idx syntax element. In the above example, lfnst_idx needs to be equal to zero to enable parsing of mts_idx.

[0235] These conditions, either separately or together, can be referred to as primary change indication conditions.

[0236] The indication of the secondary transform mode lfnst_idx can be done within one transform block, but the effect of the indication can cover multiple transform blocks. For example, in the case of a single-tree codec unit containing luma and chroma transform blocks, the secondary transform availability condition of the luma transform block can be used to determine lfnst_idx for the luma transform block. The indicated lfnst_idx can then also be used as the secondary transform mode for the chroma transform blocks within the same codec unit or transform unit.

[0237] In case of dual-tree codec where the luma component is separated into its own codec structure and the chroma components are separated into another codec structure, the lfnst_idx of the luma transform block can be determined based on the secondary transform availability conditions for that luma block. When chroma transform blocks are involved, a joint lfnst_idx can be indicated for both chroma blocks by including the secondary transform availability conditions in the first chroma transform block with non-zero transform coefficients. This can be implemented, for example, using the following additional conditions in the generic secondary transform availability conditions lfnstGenericConditions:

[0238]

[0239]

[0240] Wherein cIdx equal to zero refers to the luma transform block, cIdx equal to one refers to the first chroma transform block, treeType equal to DUAL_TREE_CHROMA refers to dual-tree coding with separate luma and chroma trees and transform blocks belonging to the chroma tree, and tu_cbf_cb equal to zero refers to the first chroma transform block having only zero-valued coefficients.

[0241] It is also possible to indicate the secondary transform mode lfnst_idx and the primary transform mode mts_idx at the codec unit level. In that case, the availability condition for the secondary transform indication may use only the last coefficient position indication lastScanPos of the luma transform block or luma block within the codec unit and omit the last coefficient position indication of the chroma transform block in the case of single-tree coding and in the case of dual-tree luma coding. In the case of dual-tree chroma coding, the secondary transform availability condition may include the last coefficient position indication lastScanPos for the first chroma transform block having non-zero coefficients.

[0242] The order in which different syntax elements are encoded and decoded may depend on higher-level signaling in the bitstream. Such signaling may occur, for example, in different parameter sets or headers in the video bitstream or outside the bitstream, and may include disabling or enabling all or some primary transform modes and secondary transform modes. For example, according to such signaling, if the secondary transform has been disabled for all or part of the bitstream, the primary transform indicator may be decoded before the last coefficient indicator; if the secondary transform has been enabled for all or part of the bitstream, the primary transform indicator may be decoded after the secondary transform indicator.

[0243] Figure 6 A dual transform process according to an approach is shown. Block A illustrates a block of residual samples. Block B illustrates a block of transform coefficients in a primary transform basis (after the primary transform in the encoding direction and before the primary transform in the decoding direction). Block C illustrates a block of transform coefficients in a secondary transform basis (after the secondary transform in the encoding direction and before the secondary transform in the decoding direction). The secondary transform is typically applied to the upper left corner of the transform coefficients, which typically corresponds to low-frequency content in the input signal. Area D in block C depicts such a dual transform area or dual transform area. However, area S in block C depicts an area that has undergone a single transform stage. Depending on the processing direction, primary transform and secondary transform here can refer to forward or inverse variants of the transform.

[0244] Figure 7 The diagram shows a double transform process with an incomplete or partial secondary transform. In this case, when the secondary transform is applied in the forward direction, the secondary transform obtains the 0and Z 1 The primary transform coefficients in the region of Z are taken as input and the inverse secondary transform is able to output non-zero values ​​in the region of D. When this secondary transform is applied in the reverse direction, the coefficients in region D are used as input and the inverse secondary transform is able to output coefficients in all regions D, Z 0 and Z 1 The coefficients in , thus filling the area of ​​the primary transform.

[0245] Secondary transforms are used for residual coding, for example in the JEM exploration model (mode-dependent non-separable secondary transform, MDNSST) disclosed in JVET-G1001 “Algorithm Description of Joint Exploration Test Model 7 (JEM 7)”, July 2017, Torino, IT, and in the VTM reference implementation of the VVC draft standard (low-frequency non-separable transform, LFNST) disclosed in JVET-N1001 “Versatile Video Coding (Draft 5)”, April 2019, Geneva, CH.

[0246] In these cases, a secondary transform is applied in a 4×4 or 8×8 low-frequency region after the primary transform in the encoder and similarly before the primary transform in the decoder. In both cases, the final set of transform coefficients includes low-frequency coefficients that have been double-transformed and high-frequency coefficients that have been single-transformed. This choice may not be ideal in all scenarios.

[0247] The method and related embodiments can be implemented in different ways. For example, the order of the operations described above can be changed, or the operations can be interleaved in different ways. In addition, different additional operations can be applied at different stages of processing. For example, there can be additional filtering, scaling, mapping or other processing applied to the final or intermediate results of the described operations. The final or intermediate results of the operations described above can also be further combined with the results of other operations.

[0248] Fig.11 A block diagram of a video decoder suitable for employing embodiments of the present invention is shown. Fig.11 The structure of a two-layer decoder is depicted, but it should be understood that the decoding operations can be similarly used in a single-layer decoder.

[0249] The video decoder 550 includes a first decoder section 552 for the base layer and a second decoder section 554 for the prediction layer. Block 556 illustrates a demultiplexer for passing information about the base layer picture to the first decoder section 552 and for passing information about the prediction layer picture to the second decoder section 554. Reference P'n represents a predicted representation of an image block. Reference D'n represents a reconstructed prediction error signal. Blocks 704, 804 illustrate a preliminary reconstructed image (I'n). Reference R'n represents a final reconstructed image. Blocks 703, 803 illustrate an inverse transform (T -1 Blocks 702 and 802 illustrate inverse quantization (Q -1 Blocks 701 and 801 illustrate entropy decoding (E -1 ). Blocks 705, 805 illustrate a reference frame memory (RFM). Blocks 706, 806 illustrate prediction (P) (inter-frame prediction or intra-frame prediction). Blocks 707, 807 illustrate filtering (F). Blocks 708, 808 may be used to combine the decoded prediction error information with the predicted base layer / prediction layer image to obtain a preliminary reconstructed image (I'n). The preliminary reconstructed and filtered base layer image may be output 709 from the first decoder section 552, and the preliminary reconstructed and filtered base layer image may be output 809 from the first decoder section 554.

[0250] In this document, a decoder should be interpreted as covering any operating unit capable of performing a decoding operation, such as a player, a receiver, a gateway, a demultiplexer and / or a decoder.

[0251] As another aspect, a device is provided, comprising: at least one processor and at least one memory, the at least one memory having code stored thereon, which, when executed by the at least one processor, causes the device to at least perform: decoding a set of transform coefficients associated with a transform block; determining whether a dual transform is applied to the transform block and whether it is affirmative; performing a first transform on the decoded set of transform coefficients and generating a first set of output values; performing a second transform using the first set of output values ​​as input to the second transform and generating a second set of output values; and using the second set of output values ​​as output of the dual transform.

[0252] Such an apparatus further comprises code stored in the at least one memory, which, when executed by the at least one processor, causes the apparatus to perform one or more of the embodiments disclosed herein.

[0253] Fig.1215 is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. Data source 1510 provides source signals in analog, uncompressed digital or compressed digital formats or any combination of these formats. Encoder 1520 may include preprocessing (such as data format conversion and / or filtering of source signals) or be connected thereto. Encoder 1520 encodes source signals into codec media bitstreams. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located in almost any type of network. Additionally, the bitstream can be received from local hardware or software. Encoder 1520 may be able to encode more than one media type (such as audio and video), or more than one encoder 1520 may be required to encode and decode source signals of different media types. Encoder 1520 may also obtain synthetically generated inputs, such as graphics and text, or it may be able to generate codec bitstreams of synthetic media. In the following, only the processing of a codec media bitstream of a media type is considered to simplify the description. However, it should be noted that a broadcast service that is usually real-time includes several streams (usually at least one audio, video and text subtitle stream). It should also be noted that the system may include many encoders, but only one encoder 1520 is shown in the figure to simplify the description without lack of generality. It should also be understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles also apply to the corresponding decoding process, and vice versa.

[0254] The codec media bitstream may be delivered to a storage device 1530. The storage device 1530 may include any type of mass storage to store the codec media bitstream. The format of the codec media bitstream in the storage device 1530 may be a basic self-contained bitstream format, or one or more codec media bitstreams may be encapsulated into a container file, or the codec media bitstream may be encapsulated into a segment format suitable for DASH (or a similar streaming system) and stored as a segment sequence. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store more than one media bitstream in a file and create file format metadata, which may also be stored in the file. The encoder 1520 or the storage device 1530 may include a file generator, or the file generator may be operably attached to the encoder 1520 or the storage device 1530. Some systems operate "live", i.e., storage is omitted and the codec media bitstream is delivered directly from the encoder 1520 to the transmitter 1540. The codec media bitstream may then be delivered to the transmitter 1540, also referred to as a server, as needed. The format used in the transmission can be a basic self-contained bitstream format, a packet stream format, a segment format suitable for DASH (or similar streaming systems), or one or more codec media bitstreams can be encapsulated into a container file. The encoder 1520, storage device 1530 and server 1540 can reside in the same physical device, or they can be included in separate devices. The encoder 1520 and server 1540 can operate with live real-time content, in which case the codec media bitstream is typically not stored permanently, but is buffered for a short period of time in the content encoder 1520 and / or in the server 1540 to smooth out processing delays, delivery delays, and changes in codec media bitrates.

[0255] The server 1540 sends the coded media bitstream using a communication protocol stack. The stack may include, but is not limited to, one or more of the following: Real-time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). When the communication protocol stack is packet-oriented, the server 1540 encapsulates the coded media bitstream into packets. For example, when RTP is used, the server 1540 encapsulates the coded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. Similarly, it should be noted that the system may include more than one server 1540, but for simplicity, the following description only considers one server 1540.

[0256] If the media content is encapsulated in a container file for storage device 1530 or for inputting data to transmitter 1540, then transmitter 1540 may include or be operably attached to a "sending file parser" (not shown in the figure). Specifically, if the container file is not sent as such, but encapsulates at least one of the included codec media bitstreams for transmission over a communication protocol, then the sending file parser locates the appropriate portion of the codec media bitstream to be transmitted over the communication protocol. The sending file parser can also help create the correct format for the communication protocol, such as packet headers and payload. The multimedia container file may contain encapsulation instructions (such as a hint track in ISOBMFF) for encapsulating at least one of the included media bitstreams on the communication protocol.

[0257] Server 1540 may or may not be connected to gateway 1550 via a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. A gateway may also or alternatively be referred to as a middlebox. For DASH, a gateway may be an edge server (of a CDN) or a web proxy. It should be noted that a system may typically include any number of gateways, etc., but for simplicity, the following description only considers one gateway 1550. Gateway 1550 may perform different types of functions, such as converting a packet stream into another communication protocol stack, merging and bifurcating data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of a forwarded stream according to existing downlink network conditions. In various embodiments, gateway 1550 may be a server entity.

[0258] The system includes one or more receivers 1560, which are generally capable of receiving the transmitted signal, demodulating the transmitted signal, and decapsulating the transmitted signal into a codec media bitstream. The codec media bitstream can be passed to a recording storage device 1570. The recording storage device 1570 can include any type of mass storage to store the codec media bitstream. The recording storage device 1570 can alternatively or additionally include a computing memory, such as a random access memory. The format of the codec media bitstream in the recording storage device 1570 can be a basic self-contained bitstream format, or one or more codec media bitstreams can be encapsulated into a container file. If there are multiple codec media bitstreams associated with each other (such as audio streams and video streams), then the container file is generally used, and the receiver 1560 includes or is attached to a container file generator, which generates a container file from the input stream. Some systems run "live", that is, the recording storage device 1570 is omitted and the codec media bitstream is passed directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recorded stream (eg, a selection of the most recent 10 minutes of the recorded stream) is maintained in the recording storage device 1570, while any earlier recorded data is discarded from the recording storage device 1570.

[0259] The codec media bitstream may be passed from the recording storage device 1570 to the decoder 1580. If there are many codec media bitstreams (such as audio streams and video streams) that are associated with each other and encapsulated in a container file or a single media bitstream is encapsulated in a container file, for example, for easier access, then a file parser (not shown in the figure) is used to decapsulate each codec media bitstream from the container file. The recording storage device 1570 or the decoder 1580 may include a file parser, or the file parser is attached to the recording storage device 1570 or the decoder 1580. It should also be noted that the system may include many decoders, but only one decoder 1570 is discussed here to simplify the description without lack of generality.

[0260] The coded media bitstream may be further processed by a decoder 1570, the output of which is one or more uncompressed media streams. Finally, a renderer 1590 may reproduce the uncompressed media streams using, for example, a loudspeaker or a display. The receiver 1560, the recording storage 1570, the decoder 1570, and the renderer 1590 may reside in the same physical device, or they may be included in separate devices.

[0261] The transmitter 1540 and / or the gateway 1550 may be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast start, and / or the transmitter 1540 and / or the gateway 1550 may be configured to select (multiple) representations for transmission. The switching between different representations may occur for a variety of reasons, such as in response to a request from the receiver 1560 or an existing condition (such as throughput) of a network transmitting a bitstream. In other words, the receiver 1560 may initiate switching between representations. The request from the receiver may be, for example, a request for a segment or sub-segment from a different representation than previously, a request for a change in the scalability layer and / or sub-layer sent, or a request for a change in a rendering device having different capabilities than previously. The request for a segment may be an HTTP GET request. The request for a field may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, to provide so-called fast start in streaming services, where the bitrate of the transmitted stream is lowered below the channel bitrate after start or random access streaming, so that playback can start immediately and a buffer occupancy level is reached that tolerates occasional packet delays and / or retransmissions. Bitrate adaptation may include multiple representation or layer up-switching and representation or layer down-switching operations occurring in various orders.

[0262] Decoder 1580 can be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast startup, and / or decoder 1580 can be configured to select (multiple) representations sent. Switching between different representations can occur for a variety of reasons, such as to achieve faster decoding operations or adapt to the bit rate sent, for example, in terms of bit rate, for existing conditions (such as throughput) of the network transmitting the bit stream. For example, if the device including decoder 1580 is multitasking and uses computing resources for other purposes other than decoding the video bit stream, faster decoding operations may be required. In another example, when playing back content at a faster rhythm than normal playback speed (e.g., twice or three times faster than the conventional real-time playback rate), faster decoding operations may be required.

[0263] In the above, some embodiments have been described with reference to and / or using the terminology of HEVC. It should be understood that the embodiments may be similarly implemented with any video encoder and / or video decoder.

[0264] In the above, where example embodiments have been described with reference to an encoder, it is to be understood that the generated bitstream and decoder may have corresponding elements therein. Similarly, where example embodiments have been described with reference to a decoder, it is to be understood that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by a decoder. For example, some embodiments have been described in connection with generating prediction blocks as part of encoding. Embodiments can similarly be implemented by generating prediction blocks as part of decoding, except that codec parameters (such as horizontal offset and vertical offset) are decoded from the bitstream rather than determined by the encoder.

[0265] The embodiments of the present invention described above describe the codec according to separate encoder and decoder devices to assist in understanding the processes involved. However, it should be appreciated that the devices, structures and operations can be implemented as a single encoder-decoder device / structure / operation. In addition, it is possible that the encoder and decoder can share some or all common elements.

[0266] Although the above examples describe embodiments of the present invention operating within a codec within an electronic device, it should be appreciated that the present invention as defined in the claims may be implemented as part of any video codec. Thus, for example, embodiments of the present invention may be implemented in a video codec that may implement video coding and decoding via fixed and wired communication paths.

[0267] Hence, the user equipment may comprise a video codec, such as the video codecs described in the above embodiments of the invention.It will be appreciated that the term user equipment is intended to cover any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device or a portable web browser.

[0268] Furthermore, elements of a public land mobile network (PLMN) may also include a video codec as described above.

[0269] In general, various embodiments of the present invention may be implemented using hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented using hardware, while other aspects may be implemented using firmware or software, which may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it should be well understood that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0270] Embodiments of the present invention may be implemented by computer software or hardware or a combination of software and hardware, the computer software being executable by a data processor of a mobile device (such as in a processor entity). Further, in this regard, it should be noted that any box of the logic flow as shown in the figure may represent program steps, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. The software may be stored on physical media such as memory chips or memory blocks implemented within a processor, magnetic media (such as hard disks or floppy disks), and optical media (such as (for example) DVDs and their data variants CDs).

[0271] The memory may be of any type suitable for the local technical environment, and the memory may be implemented using any appropriate data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment, and may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture as non-limiting examples.

[0272] Embodiments of the present invention may be practiced in various components, such as integrated circuit modules. The design of integrated circuits is essentially a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0273] Programs, such as those provided by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and position components on a semiconductor chip. Once the design of a semiconductor circuit has been completed, the resulting design, in a standardized electronic format (e.g., Opus, GDSII, etc.), may be sent to a semiconductor manufacturing facility or "fab" for fabrication.

[0274] The foregoing description has provided by way of exemplary and non-limiting examples a complete and informative description of exemplary embodiments of the present invention. However, various modifications and alterations may become apparent to those skilled in the relevant arts in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. Nevertheless, all such modifications and similar modifications of the teachings of the present invention will still fall within the scope of the present invention.

Claims

1. An apparatus for processing a video, comprising components for: obtaining a block of transform coefficients; searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order; selecting a secondary transform mode based on the position of the last non-zero coefficient, wherein the secondary transform mode corresponds to a secondary transform, and encoding a first syntax element indicating the selected secondary transform mode; selecting a primary transform mode based on the selected secondary transform mode, wherein a primary transform corresponds to the primary transform mode, and encoding a second syntax element indicating the selected primary transform mode, wherein the primary transform is applied to one region of the block of transform coefficients and both the primary transform and the secondary transform are applied to another region of the block of transform coefficients; encoding a third syntax element defining the transform coefficient within the block of transform coefficients; and sending the first, second, and third syntax elements as part of a bitstream, wherein an indication of the primary transform mode is in the bitstream when the secondary transform mode is zero, and an indication of the primary transform mode is not in the bitstream when the secondary transform mode is non-zero, wherein the primary transform is implemented as a discrete cosine transform (DCT) when the secondary transform is used, and wherein a low-frequency non-separate transform is used as the secondary transform coupled with the DCT used as the primary transform when the secondary transform mode is non-zero.

2. The apparatus according to claim 1, further comprising means for: The first syntax element indicating the secondary transform mode is encoded based on the position of the last non-zero coefficient being within a predetermined range of the scanning order in the block.

3. The apparatus according to claim 1 , further comprising means for selecting the primary conversion mode or the secondary conversion mode by: Test different transformation modes; evaluating a cost function for the different transformation modes tested; and The transformation mode is selected that minimizes the output of the cost function.

4. The apparatus according to claim 1, further comprising means for: Signaling of at least a portion of syntax elements related to the primary transform mode is omitted in case a non-zero secondary transform mode is included in the bitstream.

5. The apparatus of claim 1 , further comprising means for: Prior to signaling the position of the last non-zero coefficient in the block of transform coefficients, a transform skip mode is signaled when both a primary transform and a secondary transform are bypassed.

6. The apparatus of claim 1, further comprising means for: signaling said position of said last non-zero coefficient; and After signaling the position of the last non-zero coefficient, the first syntax element indicating the selected secondary transform mode is signaled.

7. The apparatus of claim 1, further comprising means for: signaling that the secondary transform is to be turned off for the block of transform coefficients; After signaling the first syntax element indicating the selected secondary transform mode, the second syntax element indicating the selected primary transform mode is signaled.

8. The device according to any one of claims 1 to 7, further comprising means for: After signaling the second syntax element indicating the selected primary transform mode, syntax elements defining the coefficients of the block of transform coefficients are signaled.

9. A method for processing a video, comprising: obtaining a block of transform coefficients; searching the block of transform coefficients for the position of the last non-zero coefficient in scanning order; selecting a secondary transform mode based on the position of the last non-zero coefficient, wherein the secondary transform mode corresponds to a secondary transform, and encoding a first syntax element indicating the selected secondary transform mode; selecting a primary transform mode based on the selected secondary transform mode, wherein a primary transform corresponds to the primary transform mode, wherein the primary transform is applied to one region of the block of transform coefficients and both the primary transform and the secondary transform are applied to another region of the block of transform coefficients, and encoding a second syntax element indicating the selected primary transform mode; encoding a third syntax element defining the transform coefficient within the block of transform coefficients; and sending the first, second, and third syntax elements as part of a bitstream, wherein an indication of the primary transform mode is in the bitstream when the secondary transform mode is zero, and an indication of the primary transform mode is not in the bitstream when the secondary transform mode is non-zero, wherein the primary transform is implemented as a discrete cosine transform (DCT) when the secondary transform is used, and wherein a low-frequency non-separate transform is used as the secondary transform coupled with the DCT used as the primary transform when the secondary transform mode is non-zero.

10. An apparatus for processing a video, comprising means for: receiving a bitstream comprising encoded information associated with a block of transform coefficients; an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream in scanning order; using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream; If an indication of a secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode; using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream; If an indication of a primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode; Decoding the transform coefficients; as well as If the determined transform mode exists, the determined transform mode is applied to select a transform, and the transform is applied to the transform coefficients.

11. The apparatus of claim 10, further comprising means for: Decoding of coefficient regions that are not applicable to the determined primary transform or the determined primary-secondary transform pair is omitted.

12. The apparatus of claim 10, further comprising means for: applying at least one of a primary transform and a secondary transform to one or more transform blocks subsequent to the block of transform coefficients, the block of transform coefficients comprising an indication indicating at least one of the primary transform mode and the secondary transform mode; and wherein said primary transformation corresponds to said primary transformation mode; Wherein the secondary transform corresponds to the secondary transform mode.

13. The apparatus of claim 10, 11 or 12, further comprising means for: The indication of a primary transform mode is decoded from the bitstream only if a predetermined condition is met.

14. The apparatus according to claim 13, wherein the predetermined condition comprises one of the following: the minimum dimension of the block of transform coefficients; The maximum dimension of the block of transform coefficients.

15. A method for processing a video, comprising: receiving a bitstream comprising encoded information associated with a block of transform coefficients; an indication of the position of the last non-zero coefficient in the block of transform coefficients decoded from the bitstream in scanning order; using the position of the last coefficient to determine whether an indication of a secondary transform mode is present in the bitstream; If an indication of a secondary transform mode is present in the bitstream, determining the secondary transform mode by decoding a first syntax element indicating the secondary transform mode; using the determined secondary transform mode to determine whether an indication of a primary transform mode is present in the bitstream; If an indication of a primary transform mode is present in the bitstream, determining the primary transform mode by decoding a second syntax element indicating the primary transform mode; Decoding the transform coefficients; as well as If the determined primary and secondary transform modes exist, the determined primary and secondary transform modes are applied to select corresponding primary and secondary transforms, wherein the primary transform is based on the secondary transform, and the primary and secondary transforms are applied to the transform coefficients, wherein the primary transform is applied to one region of the block of transform coefficients, and both the primary transform and the secondary transform are applied to another region of the block of transform coefficients.

Images