Apparatus, method and computer program for video encoding and decoding

Abstract

A method includes: obtaining a configuration of at least one neural network including multiple intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers, the one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; obtaining at least one input video frame including multiple blocks; determining to encode one or more blocks using intra-prediction; determining an intra-prediction mode for each of the one or more blocks; grouping blocks having the same intra-prediction mode into groups, each group being assigned a computational path among the multiple intra-prediction mode-agnostic layers and the one or more intra-prediction mode-specific layers; training the multiple intra-prediction mode-agnostic layers and / or the one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network related to the block group and an annotated block, wherein the annotated block is a block of the input video frame or a reconstructed block; and encoding the block using the computational path of the intra-prediction mode assigned to the block.

Description

Technical Field

[0001] The present invention relates to apparatus, methods and computer programs for video encoding and decoding. Background Art

[0002] So-called hybrid video compression is based on performing intra-frame prediction and inter-frame prediction and then transform coding the residual or prediction error. In intra-frame mode, blocks of a video frame are predicted from spatially neighboring blocks of the same frame. Then, only the prediction error or residual between the intra-frame predicted block and the original block needs to be encoded. The intra-frame prediction process is performed not only at the decoder, but also in the encoder, more specifically in the "decoding loop" of the encoder, in order to calculate the residual.

[0003] The intra prediction process is usually performed using an algorithmic approach (i.e., without using a learning approach). Intra prediction usually involves a number of predefined prediction modes, wherein during encoding, for a block to be predicted, all or several prediction modes are tried and the most suitable prediction mode for the block is selected, for example, based on the mean square error (MSE) between the predicted block and the true block.

[0004] The goal of the intra prediction process is to obtain the minimum possible prediction error or residual, thus requiring fewer bits to encode the intra frames which consume a large portion of the bitstream of the entire video. However, the algorithmic approaches described above have been limited to this ultimate research and development which have saturated in terms of prediction accuracy. Therefore, further development can be achieved using learning methods. Summary of the invention

[0005] Now in order to at least alleviate the above problems, an enhanced encoding method is introduced here.

[0006] The method according to the first aspect includes: obtaining a configuration of at least one neural network including multiple intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers, the one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; obtaining at least one input video frame including multiple blocks; determining to encode one or more blocks using intra-prediction; determining an intra-prediction mode for each of the one or more blocks; grouping blocks having the same intra-prediction mode into groups, each group being assigned a calculation path among the multiple intra-prediction mode-agnostic layers and the one or more intra-prediction mode-specific layers; training the multiple intra-prediction mode-agnostic layers and / or the one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network related to the block group and a labeled (ground-truth) block, wherein the labeled block is a block of the input video frame or a reconstructed block; and encoding the block using the calculation path assigned to the intra-prediction mode for the block.

[0007] A second aspect relates to an apparatus comprising: a component for obtaining a configuration of at least one neural network including a plurality of intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers, the one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; a component for obtaining at least one input video frame including a plurality of blocks; a component for determining to encode the one or more blocks using intra-prediction; an apparatus for determining an intra-prediction mode for each of the one or more blocks; a component for grouping blocks having the same intra-prediction mode into groups, each group being assigned a computational path among the plurality of intra-prediction mode-agnostic layers and the one or more intra-prediction mode-specific layers; a component for training the plurality of intra-prediction mode-agnostic layers and / or the one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network related to the block group and an annotated block, wherein the annotated block is a block of the input video frame or a reconstructed block; and a component for encoding the block using the computational path assigned to the intra-prediction mode for the block.

[0008] According to an embodiment, the configuration of at least one neural network comprises a pre-trained neural network, and the apparatus further comprises means for using blocks of an input video frame as labeled blocks.

[0009] According to an embodiment, the apparatus further comprises means for indicating in or along a bitstream comprising the coding block groups information indicating the block groups defined as training blocks.

[0010] According to an embodiment, a configuration of a neural network includes a mode-agnostic region of (one or more) layers and a mode-specific region of (one or more) layers.

[0011] According to an embodiment, the configuration of the neural network includes an initial region of initial layers trained for multiple intra prediction modes, and multiple final layers specific to each intra prediction mode.

[0012] According to an embodiment, the configuration of the neural network includes a plurality of different neural networks for different intra prediction modes.

[0013] According to an embodiment, a first learning rate is used for training an intra prediction mode agnostic layer, a second learning rate is used for training an intra prediction mode specific layer, and wherein the second learning rate is smaller than the first learning rate.

[0014] According to an embodiment, the information indicating the block group defined as the training block includes a quantization parameter, wherein if the quantization parameter is less than or equal to a threshold value, the block group is defined as the training block.

[0015] According to an embodiment, the input to the neural network includes a plurality of reconstructed blocks of an input video frame.

[0016] According to an embodiment, epochs or iterations for training a plurality of intra prediction mode agnostic layers and / or one or more intra prediction mode specific layers are based on blocks associated with a plurality of intra prediction modes.

[0017] A third aspect relates to a method comprising: obtaining a configuration of at least one neural network comprising a plurality of intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; obtaining a bitstream comprising an encoded video frame, wherein the encoded video frame comprises a plurality of blocks; decoding information indicating a block group defined as a training block from or along the bitstream; decoding the training block into a reconstructed block; training the plurality of intra-prediction mode-agnostic layers and / or the one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network related to the block group and the reconstructed block; and decoding the encoded block using a calculation path of the intra-prediction mode assigned to the encoded block.

[0018] A fourth aspect relates to an apparatus comprising: a component for obtaining a configuration of at least one neural network including a plurality of intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; a component for obtaining a bitstream including an encoded video frame, wherein the encoded video frame includes a plurality of blocks; a component for decoding information indicating a block group defined as a training block from or along the bitstream; a component for decoding the training block into a reconstructed block; a component for training a plurality of intra-prediction mode-agnostic layers and / or one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network related to the block group and the reconstructed block; and a component for decoding the encoded block using a calculation path of the intra-prediction mode assigned to the encoded block.

[0019] The apparatus according to the fifth and sixth aspects comprises at least one processor and at least one memory on which code is stored, the at least one memory and the computer program code being configured to, together with the at least one processor, cause the apparatus to perform a method.

[0020] According to yet another aspect, a computer-readable non-transitory storage medium having stored thereon code for use by an apparatus, the code when executed by a processor causing the apparatus to perform a method is provided.

[0021] Therefore, the apparatus and the computer-readable storage medium having code stored thereon as described above are arranged to perform one or more of the above-described methods and embodiments related thereto.

[0022] A further aspect relates to an apparatus comprising means for performing one or more of the above-described methods and embodiments related thereto. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings, in which:

[0024] Figure 1 Schematically illustrates an electronic device using an embodiment of the present invention;

[0025] Figure 2 Schematically illustrating a user equipment suitable for adopting an embodiment of the present invention;

[0026] Figure 3 Further schematically illustrated are electronic devices employing embodiments of the present invention connected using wireless and wireless network connections;

[0027] Figure 4 A block diagram schematically shows a general encoder;

[0028] Figure 5 shows an example of an intra prediction process for a block to be predicted;

[0029] Figure 6 A flowchart showing an encoding method according to an embodiment of the present invention;

[0030] Figure 7 A flowchart showing a decoding method according to an embodiment of the present invention;

[0031] Figure 8 A schematic diagram showing a neural network for intra-frame prediction according to an embodiment of the present invention;

[0032] Fig. 9 An example of determining a label calculation path label for a block of a frame according to an embodiment of the present invention is shown;

[0033] Fig.10 An example of grouping blocks having the same annotated computation path label according to an embodiment of the present invention is shown; and

[0034] Fig.11 A schematic diagram is shown of an example multimedia communication system within which various embodiments may be implemented. DETAILED DESCRIPTION

[0035] Suitable devices and possible mechanisms for neural network-based video / image coding are described in more detail below. In this regard, reference is first made to Figure 1 and Figure 2 ,in, Figure 1 A block diagram of a video encoding system according to an example embodiment is shown as a schematic block diagram of an exemplary apparatus or electronic device 50 that may include a codec according to an embodiment of the present invention. Figure 2The layout of the device according to the exemplary embodiment is shown. Then the Figure 1 and Figure 2 components.

[0036] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be understood that embodiments of the present invention may be implemented in any electronic device or apparatus that may require encoding and decoding or encoding or decoding video images.

[0037] The device 50 may include a housing 30 for containing and protecting the device. The device 50 may also include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may also include a keypad 34. In other embodiments of the present invention, any suitable data or user interface mechanism may be used. For example, the user interface may be implemented as a virtual keyboard or data input system as part of a touch-sensitive display.

[0038] The device may include a microphone 36 or any suitable audio input that may be a digital or analog signal input. The device 50 may also include an audio output device, which in embodiments of the present invention may be any of the following: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other embodiments of the present invention, the device may be powered by any suitable mobile energy device (such as a solar cell, a fuel cell, or a clockwork generator)). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as, for example, a Bluetooth wireless connection or a USB / Firewire wired connection.

[0039] The apparatus 50 may include a controller 56, a processor or processor circuit for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which, in embodiments of the invention, may store data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuit 54 which is suitable for performing encoding and / or decoding of audio and / or video data or assisting in encoding and decoding performed by the controller.

[0040] The apparatus 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and adapted to provide authentication information for authenticating and authorizing the user at the network.

[0041] The device 50 may include a radio interface circuit 52 connected to the controller and suitable for generating wireless communication signals, for example, for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may also include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated at the radio interface circuit 52 to other (one or more) devices and for receiving radio frequency signals from other (one or more) devices.

[0042] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or a controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may also receive images wirelessly or via a wired connection for encoding / decoding. The structural elements of the device 50 described above represent examples of components for performing corresponding functions. It should be understood that although some embodiments have been described using the codec 54 as an example, the video encoding device according to the embodiment may include (one or more) encoders or (one or more) decoders, or both.

[0043] about Figure 3 , showing an example of a system in which embodiments of the present invention may be utilized. System 10 includes a plurality of communication devices that may communicate via one or more networks. System 10 may include any one or a combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) such as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, broadcast networks, and the Internet.

[0044] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the present invention.

[0045] For example, Figure 3 The illustrated system shows a representation of the mobile telephone network 11 and the Internet 28. Connections to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections (including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths).

[0046] The example communication devices shown in system 10 may include, but are not limited to, electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and mobile phone 14, PDA 16, integrated messaging device (IMD) 18, desktop computer 20, notebook computer 22. Apparatus 50 may be stationary or mobile when carried by an individual who is moving. Apparatus 50 may also be located in a mode of transportation, including, but not limited to, a car, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or any similarly suitable mode of transportation.

[0047] Embodiments may also be implemented in a set-top box (i.e., a digital TV receiver that may or may not have a display or wireless capabilities), in a tablet or (laptop) personal computer (PC) having hardware or software or a combination of encoder / decoder implementations, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems that provide hardware / software based encoding.

[0048] Some or further devices may send and receive calls and messages and communicate with a service provider via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0049] The communication device may communicate using various transmission technologies, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short message service (SMS), multimedia message service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. The communication device involved in implementing various embodiments of the present invention may communicate using various media, including but not limited to radio, infrared, laser, cable connection and any suitable connection.

[0050] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wire, while a logical channel can refer to a logical connection on a multiplexed medium capable of transmitting several logical channels. A channel can be used to transmit information signals, such as a bit stream, from one or several senders (or transmitters) to one or several receivers.

[0051] The MPEG-2 Transport Stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video and other media, as well as program metadata or other metadata in a multiplexed stream. A packet identifier (PID) is used to identify elementary streams (also called packetized elementary streams) within a TS. Therefore, logical channels within an MPEG-2 TS can be considered to correspond to specific PID values.

[0052] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and a file format for NAL unit structured video derived from ISOBMFF (ISO / IEC 14496-15).

[0053] A video codec may include an encoder that transforms an input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a viewable form. The video encoder and / or video decoder may also be separate from each other, i.e., need not form a codec. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0054] A typical hybrid video encoder, such as many encoder implementations of ITU-T H.263 and H.264, encodes video information in two stages. First, the pixel values ​​in a particular picture region (or "block") are predicted, for example by a motion compensation component (finding and indicating an area in a previously encoded video frame that closely corresponds to the block being encoded) or by a spatial component (using the pixel values ​​surrounding the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0055] In temporal prediction, the prediction source is a previously decoded picture (also called a reference picture). In intra-block copy (IBC; also called intra-block copy prediction), prediction is applied similarly to temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be similarly applied to temporal prediction, but the reference picture is a picture decoded from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer to only temporal prediction, while in other cases, inter-frame prediction may collectively refer to temporal prediction and any of the following: intra-block copy, inter-layer prediction, and inter-view prediction, provided that they are performed with the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0056] Inter prediction (also known as temporal prediction, motion compensation, or motion compensated prediction) reduces temporal redundancy. In inter prediction, the prediction source is a previously decoded picture. Intra prediction exploits the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in the spatial domain or in the transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is typically used in intra coding, where inter prediction is not applied.

[0057] One result of the encoding process is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially neighboring motion vectors, and only the difference with respect to the motion vector predictor can be encoded. Prediction of coding parameters and intra-frame prediction may be collectively referred to as intra-picture prediction.

[0058] Figure 4 Block diagram showing the general structure of a video encoder. Figure 4 An encoder for two layers is presented, but it will be appreciated that the presented encoder may be similarly extended to encode more than two layers or to encode a single layer. Figure 4 The video encoder shown includes a first encoder part 500 for the base layer and a second encoder part 502 for the enhancement layer. Each of the first encoder part 500 and the second encoder part 502 may include similar elements for encoding an incoming picture. The encoder parts 500, 502 may include a pixel predictor 302, 402, a prediction error encoder 303, 403 and a prediction error decoder 304, 404. Figure 4Also shown are embodiments of pixel predictors 302, 402 including inter-frame predictors 306, 406, intra-frame predictors 308, 408, mode selectors 310, 410, filters 316, 416, and reference frame memories 318, 418. The pixel predictor 302 of the first encoder portion 500 receives the base layer image 300 of the video stream to be encoded at both the inter-frame predictor 306 (which determines the difference between the image and the motion compensated reference frame 318) and the intra-frame predictor 308 (which determines the prediction of the image block based only on the processed portion of the current frame or picture). The outputs of the inter-frame predictor and the intra-frame predictor are both passed to the mode selector 310. The intra-frame predictor 308 can have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide a prediction signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer picture 300. Accordingly, the pixel predictor 402 of the second encoder portion 502 receives the enhancement layer image 400 of the video stream to be encoded at both the inter-frame predictor 406 (which determines the difference between the image and the motion compensated reference frame 418) and the intra-frame predictor 408 (which determines the prediction of the image block based only on the processed portion of the current frame or picture). The outputs of the inter-frame predictor and the intra-frame predictor are both passed to the mode selector 410. The intra-frame predictor 408 can have more than one intra-frame prediction mode. Therefore, each mode can perform intra-frame prediction and provide a prediction signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer picture 400.

[0059] Depending on which coding mode is selected to encode the current block, the output of the inter-frame predictor 306, 406 or the output of one of the optional intra-frame predictor modes or the output of the surface encoder in the mode selector is passed to the output of the mode selector 310, 410. The output of the mode selector is passed to the first summing device 321, 421. The first summing device can subtract the output of the pixel predictor 302, 402 from the base layer picture 300 / enhancement layer picture 400 to produce a first prediction error signal 320, 420, which is input to the prediction error encoder 303, 403.

[0060] The pixel predictor 302, 402 further receives a combination of a predicted representation of the image block 312, 412 and an output 338, 438 of the prediction error decoder 304, 404 from the preliminary reconstructor 339, 439. The preliminary reconstructed image 314, 414 may be passed to the intra predictor 308, 408 and the filter 316, 416. The filter 316, 416 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image 340, 440 which may be stored in a reference frame memory 318, 418. The reference frame memory 318 may be connected to the inter predictor 306 for use as a reference image for comparison with the future base layer picture 300 in an inter prediction operation. According to some embodiments, subject to the base layer being selected and indicated as a source for inter-layer sample prediction and / or inter-layer motion information prediction of the enhancement layer, the reference frame memory 318 may also be connected to the inter-frame predictor 406 to be used as a reference image for comparison with the future enhancement layer picture 400 in the inter-frame prediction operation. In addition, the reference frame memory 418 may be connected to the inter-frame predictor 406 to be used as a reference image for comparison with the future enhancement layer picture 400 in the inter-frame prediction operation.

[0061] According to some embodiments, the filter parameters of the filter 316 from the first encoder portion 500 may be provided to the second encoder portion 502 subject to the base layer being selected and indicated as the source of filter parameters for predicting the enhancement layer.

[0062] The prediction error encoder 303, 403 comprises a transform unit 342, 442 and a quantizer 344, 444. The transform unit 342, 442 transforms the first prediction error signal 320, 420 into a transform domain. The transform is, for example, a DCT transform. The quantizer 344, 444 quantizes the transform domain signal (eg, DCT coefficients) to form quantized coefficients.

[0063] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs the reverse process of the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438, which when combined with the predicted representation of the image block 312, 412 at the second summing device 339, 439 produces the preliminary reconstructed image 314, 414. The prediction error decoder can be considered to include: a dequantizer 361, 461, which dequantizes the quantized coefficient values ​​(e.g. DCT coefficients) to reconstruct the transformed signal; and an inverse transform unit 363, 463, which performs an inverse transform on the reconstructed transformed signal, wherein the output of the inverse transform unit 363, 463 contains (one or more) reconstructed blocks. The prediction error decoder may also include a block filter, which may filter (one or more) reconstructed blocks according to further decoded information and filter parameters.

[0064] The entropy encoder 330, 430 receives the output of the prediction error encoder 303, 403 and may perform suitable entropy encoding / variable length encoding on the signal to provide error detection and correction capabilities. The output of the entropy encoder 330, 430 may be inserted into the bitstream, for example, via a multiplexer 508.

[0065] Entropy encoding / decoding can be performed in many ways. For example, context-based encoding / decoding can be applied, in which both the encoder and the decoder modify the context state of the encoding parameters based on the previously encoded / decoded encoding parameters. Context-based encoding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Alternatively or additionally, entropy encoding / decoding can be performed using a variable length coding scheme, such as Huffman encoding / decoding or Exp-Golomb encoding / decoding. Decoding encoding parameters from an entropy-coded bitstream or codeword can be referred to as parsing.

[0066] The H.264 / AVC standard was developed by the Joint Video Group (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector (ITU-T) of the International Telecommunication Union and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was issued by these two previous standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are already multiple versions of the H.264 / AVC standard, which integrate new extensions or features into the specification. These extensions include scalable video coding (SVC) and multi-view video coding (MVC).

[0067] Version 1 of the High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaboration Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by these two previous standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include scalable, multi-view, fidelity range extension, three-dimensional, and screen content coding extensions, which can be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0068] SHVC, MV-HEVC, and 3D-HEVC use a common base specification specified in Annex F of Version 2 of the HEVC standard. This common base includes, for example, high-level syntax and semantics, which, for example, specifies some characteristics of the layers of the bitstream (such as inter-layer dependencies) and decoding processes (such as reference picture list construction, which includes inter-layer reference pictures and picture order count derivation for multi-layer bitstreams). Annex F may also be used for potential subsequent multi-layer extensions of HEVC. It should be understood that even though video encoders, video decoders, encoding methods, decoding methods, bitstream structures, and / or embodiments may be described below with reference to specific extensions (such as SHVC and / or MV-HEVC), they are generally applicable to any multi-layer extension of HEVC, and more generally to any multi-layer video coding scheme.

[0069] Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC and HEVC are described in this section as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures, in which embodiments can be implemented. Some of the key definitions, bitstream and coding structures, and concepts of H.264 / AVC are the same as those in HEVC, so they are described jointly below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but are described for one possible basis on which the present invention can be partially or fully implemented.

[0070] Similar to many earlier video coding standards, bitstream syntax and semantics as well as a decoding process for error-free bitstreams are specified in H.264 / AVC and HEVC. The encoding process is not specified, but the encoder must produce a conforming bitstream. Conformance of the bitstream and the decoder can be verified with a hypothetical reference decoder (HRD). The standards contain coding tools that help deal with transmission errors and losses, but the use of the tools in encoding is optional, and the decoding process for erroneous bitstreams is not yet specified.

[0071] The basic unit for the input of an H.264 / AVC or HEVC encoder and the output of an H.264 / AVC or HEVC decoder, respectively, is a picture. A picture given as input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture.

[0072] The source picture and the decoded picture each include one or more sample arrays, such as one of the following sets of sample arrays:

[0073] - Luminance (Y) only (monochrome),

[0074] - Luma and two chrominances (YCbCr or YCgCo),

[0075] - Green, Blue, and Red (GBR, also known as RGB),

[0076] - An array representing otherwise unspecified monochrome or tri-stimulus color sampling (e.g., YZX, also called XYZ).

[0077] In the following, these arrays may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation used. The actual color representation used may be indicated, for example, in the coded bitstream using the Video Usability Information (VUI) syntax of H.264 / AVC and / or HEVC. A component may be defined as an array or a single sample from one of the three sample arrays (luminance and two chroma) or an array or a single sample of an array constituting a picture in a monochrome format.

[0078] In H.264 / AVC and HEVC, a picture can be a frame or a field. A frame consists of a matrix of luma samples and possible corresponding chroma samples. When the source signal is interlaced, a field is a set of alternating sample rows of a frame and can be used as encoder input. When compared to the luma sample array, the chroma sample array may not exist (and therefore, monochrome sampling may be used) or the chroma sample array may be subsampled. The chroma format can be summarized as follows:

[0079] - In monochrome sampling, there is only one sample array, which can be nominally considered as a brightness array;

[0080] - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array;

[0081] - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array;

[0082] - In 4:4:4 sampling, when separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0083] In H.264 / AVC and HEVC, sample arrays can be encoded into the bitstream as separate color planes, and the separately encoded color planes can be decoded separately from the bitstream. When separate color planes are used, each color plane is processed separately (encoder and / or decoder) as a picture with monochrome sampling.

[0084] Partitioning can be defined as dividing a set into subsets such that every element of the set is in exactly one subset.

[0085] When describing the operation of HEVC encoding and / or decoding, the following terms may be used. A coding block may be defined as a block of N×N samples for a certain value of N, so that the partitioning of a coding tree block into coding blocks is a partition. A coding tree block (CTB) may be defined as a block of N×N samples for a certain value of N, so that the partitioning of a component into coding tree blocks is a partition. A coding tree unit (CTU) may be defined as a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples of a picture with three sample arrays, or a coding tree block of samples of a monochrome picture or a picture encoded using three separate color planes and a syntax structure for encoding samples. A coding unit (CU) may be defined as a coding block of luma samples, two corresponding coding blocks of chroma samples of a picture with three sample arrays, or a coding block of samples of a monochrome picture or a picture encoded using three separate color planes and a syntax structure for encoding samples. A CU with a maximum allowed size may be named an LCU (maximum coding unit) or a coding tree unit (CTU), and a video picture is divided into non-overlapping LCUs.

[0086] A CU comprises one or more prediction units (PUs) defining a prediction process for samples within the CU and one or more transform units (TUs) defining a prediction error coding process for samples in the CU. Typically, a CU comprises a square block of samples having a size selectable from a predefined set of possible CU sizes. Each PU and TU can be further partitioned into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it that defines what kind of prediction to apply to the pixels within the PU (e.g., motion vector information for PUs for inter-prediction and intra-prediction direction information for PUs for intra-prediction).

[0087] Each TU may be associated with information describing the prediction error decoding process for the samples within the TU, including, for example, DCT coefficient information. Typically, it is signaled at the CU level whether prediction error coding is applied to each CU. In the absence of a prediction error residual associated with a CU, it may be considered that there is no TU for the CU. The partitioning of an image into CUs and the partitioning of a CU into PUs and TUs is typically signaled in a bitstream that allows a decoder to reproduce the expected structure of these units.

[0088] In some coding systems, such as HEVC, a picture can be divided into tiles, which are rectangular and contain an integer number of blocks, such as LCUs in HEVC. In HEVC, the division into tiles forms a regular grid that can be represented by a list of tile column widths and a list of tile row heights. The tiles are ordered consecutively in the bitstream in the raster scan order of the tile grid. A tile may contain an integer number of slices, or a slice may contain an integer number of tiles. Blocks (such as CTUs in HEVC) can be scanned tile by tile in encoding and decoding in the raster scan order of the blocks, and the tiles can be scanned in raster scan order along the tile grid.

[0089] Video coding standards and specifications may allow encoders to divide coded pictures into coded slices, etc. Intra-picture prediction cannot usually be performed across slice boundaries. Therefore, slices can be considered as a way to divide coded pictures into independently decodable pieces. Therefore, slices are often considered to be the basic unit for transmission. In many cases, the encoder can indicate in the bitstream which types of intra-picture prediction are turned off across slice boundaries, and the decoder operation takes this information into account, for example, when inferring which prediction sources are available. For example, if a neighboring block resides in a different slice, the samples from the neighboring block can be considered unavailable for intra-frame prediction.

[0090] In HEVC, a slice is defined as an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are ordered consecutively in a block scan and contained in a single NAL unit. The division of each picture into slice segments is partitioning. In HEVC, an independent slice segment is defined as a slice segment whose values ​​of syntax elements of a slice segment header are not inferred from the values ​​of a previous slice segment, and a dependent slice segment is defined as a slice segment whose values ​​of some syntax elements of a slice segment header are inferred from the values ​​of a previous independent slice segment in decoding order. In HEVC, a slice header is defined as a slice segment header of an independent slice segment, where the independent slice segment is the current slice segment or an independent slice segment before the current dependent slice segment, and a slice segment header is defined as a portion of a coded slice segment that contains data elements belonging to the first or all coding tree units represented in the slice segment. If the block is not used, the CU is scanned in the raster scan order of the LCU within the block or within the picture. Within the LCU, the CU has a specific scanning order.

[0091] In wavefront parallel processing (WPP), each block row of a slice (e.g., a CTU row in HEVC) can be encoded and decoded in parallel. When using WPP, the state of the entropy codec at the beginning of a block row is obtained from the state of the entropy codec of the block row above after processing the second block of the row. Therefore, block rows can be processed in parallel with a delay of 2 blocks per block row.

[0092] The motion constraint block set (MCTS) constrains the inter prediction process in encoding so that sample values ​​outside the motion constraint block set and sample values ​​at fractional sample positions derived using one or more sample values ​​outside the motion constraint block set are not used for inter prediction of any sample within the motion constraint block set. In addition, the encoding of the MCTS is constrained in a manner that motion vector candidates are not derived from blocks outside the MCTS. This can be enforced by turning off the temporal motion vector prediction of HEVC or by not allowing the encoder to use the TMVP candidate in the merge or AMVP candidate list or any motion vector prediction candidate after the TMVP candidate for the PU directly to the left of the right block boundary of the MCTS but except for the last one in the lower right corner of the MCTS. In general, the MCTS can be defined as a block set that is independent of any sample values ​​and coded data (such as motion vectors) outside the MCTS. In some cases, the MCTS may be required to form a rectangular area. It should be understood that, depending on the context, the MCTS may refer to a block set within a picture or to a corresponding block set in a picture sequence. The corresponding block set can, but generally does not need to be concatenated in a picture sequence.

[0093] Note that slices, wavefronts, and tiles, when included in MCTS, can also be used as a parallelization tool that enables running several encoding and / or decoding threads or processes in parallel.

[0094] Note that sample positions used in inter prediction may be saturated by the encoding and / or decoding process such that positions outside a picture that would otherwise be saturated to point to corresponding boundary samples of the picture. Thus, if a block boundary is also a picture boundary, in some use cases the encoder may allow motion vectors to effectively cross that boundary or allow motion vectors to effectively cause fractional sample interpolation that would reference positions outside that boundary because the sample positions are saturated to the boundary. In other use cases, specifically if a coded block can be extracted from a bitstream where the block is located adjacent to a picture boundary to another bitstream where the block is located not adjacent to a picture boundary, the encoder may constrain motion vectors on picture boundaries, similar to any MCTS boundaries.

[0095] The temporal motion constrained blockset SEI message of HEVC may be used to indicate the presence of motion constrained blocksets in the bitstream.

[0096] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of the pixel block (using motion or spatial information created by the encoder and stored in a compressed representation) and prediction error decoding (the inverse operation of prediction error coding to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction component and the prediction error decoding component, the decoder adds the prediction signal and the prediction error signal (pixel value) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing the video for display and / or storing the video as a prediction reference for upcoming frames in the video sequence.

[0097] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF).H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0098] The deblocking loop filter may include multiple filtering modes or strengths, which may be adaptively selected based on characteristics of blocks adjacent to the boundary (such as quantization parameter values) and / or signaling included in the bitstream by the encoder. For example, the deblocking loop filter may include a normal filtering mode and a strong filtering mode, which may differ in the number of filter taps (i.e., the number of samples filtered on both sides of the boundary) and / or the filter tap values. For example, when ignoring the potential impact of the clipping operation, the filtering of two samples along both sides of the boundary may be performed with a filter having an impulse response of (3 7 9-3) / 16.

[0099] Next, an example of SAO is given with reference to HEVC; however, SAO can also be similarly applied to other coding schemes. In SAO, a picture is divided into regions, where a separate SAO decision is made for each region. In HEVC, the basic unit for adapting SAO parameters is the CTU (thus, an SAO region is a block covered by the corresponding CTU).

[0100] In the SAO algorithm, samples in a CTU are classified according to a set of rules, and each classified set of samples is enhanced by adding an offset value. The offset value is signaled in the bitstream. There are two types of offsets: 1) band offset, 2) edge offset. For CTU, SAO or band offset or edge offset is not used. The choice of whether to not use SAO or band or edge offset can be decided by the encoder using, for example, rate-distortion optimization (RDO) and signaled to the decoder.

[0101] In band offset, the entire range of sample values ​​is divided into 32 bands of equal width in some embodiments. For example, for 8-bit samples, the width of the band is 8 (= 256 / 32). From the 32 bands, 4 of them are selected, and for each of the selected bands, a different offset is signaled. The selection decision is made by the encoder and can be signaled as follows: the index of the first band is signaled, and then it is inferred that the following four bands are the selected bands. Band offset can be useful in correcting errors in smooth areas.

[0102] In the edge offset type, the edge offset (EO) type can be selected from four possible types (or edge classifications), where each type is associated with a direction: 1) vertical, 2) horizontal, 3) 135 degree diagonal, and 4) 45 degree diagonal. The choice of direction is made by the encoder and sent to the decoder with a signal. Each type defines the position of two neighboring samples of a given sample based on an angle. Then, each sample in the CTU is classified into one of five categories based on the comparison of the sample value with the values ​​of the two neighboring samples. After each sample in the edge offset type CTU is classified into one of the five categories, the offset value for each of the first four categories is determined and sent to the decoder with a signal. The offset for each category is added to the sample value associated with the corresponding category. Edge offset can be effective in correcting ringing effects.

[0103] Adaptive loop filter (ALF) is another method to enhance the quality of reconstructed samples. This can be achieved by filtering the sample values ​​in the loop. In some embodiments, the encoder determines which area of ​​the picture to be filtered and the filter coefficients based on, for example, RDO, and this information is signaled to the decoder.

[0104] In a typical video codec, motion information is indicated by a motion vector associated with each motion compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement of a prediction source block in an image block to be encoded (on the encoder side) or decoded (on the decoder side) and in one of the previously encoded or decoded pictures. In order to efficiently represent the motion vectors, these motion vectors are usually differentially encoded relative to the block-specific predicted motion vector. In a typical video codec, the predicted motion vector is created in a predefined manner, such as calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to create a motion vector prediction is to generate a list of candidate predictions based on adjacent blocks and / or co-located blocks in a temporal reference picture, and signal the selected candidate as a motion vector predictor. In addition to predicting the motion vector value, it is possible to predict which reference picture(s) are used for motion compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously encoded / decoded picture. The reference index is usually predicted based on adjacent blocks and / or co-located blocks in a temporal reference picture. Furthermore, a typical high efficiency video codec employs an additional motion information encoding / decoding mechanism (commonly referred to as merge / merge mode), in which all motion field information including motion vectors and corresponding reference picture indexes for each available reference picture list is predicted and used without any modification / correction. Similarly, predicting motion field information is performed using motion field information of neighboring blocks and / or co-located blocks in a temporal reference picture, and the used motion field information is signaled among the list of motion field candidate lists populated with motion field information of available neighboring / co-located blocks.

[0105] In a typical video codec, the prediction residuals after motion compensation are first transformed with a transform kernel (like DCT) and then encoded. The reason for this is that there is often still some correlation among the residuals, and in many cases the transform can help reduce this correlation and provide more efficient coding.

[0106] Typical video encoders use a Lagrangian cost function to find the best coding mode, such as the desired coding mode for a block and associated motion vectors. Such a cost function uses a weight factor λ to connect the (exact or estimated) image distortion due to the lossy coding method and the (exact or estimated) amount of information required to represent the pixel values ​​in the image region:

[0107] C=D+λR (1)

[0108] Where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean square error) taking into account the mode and motion vectors, and R is the number of bits required to represent the required data to reconstruct the image block in the decoder (including the amount of data representing the candidate motion vectors).

[0109] Video coding standards and specifications may allow the encoder to divide a coded picture into coded slices, etc. Intra-picture prediction cannot generally be performed across slice boundaries. Therefore, a slice can be considered a way to partition a coded picture into independently decodable slices. In H.264 / AVC and HEVC, intra-picture prediction cannot be performed across slice boundaries. Therefore, a slice can be considered a way to partition a coded picture into independently decodable slices, and therefore, a slice is often considered a basic unit for transmission. In many cases, the encoder can indicate in the bitstream which types of intra-picture prediction are turned off across slice boundaries, and the decoder operation takes this information into account, for example, when inferring which prediction sources are available. For example, if adjacent CUs reside in different slices, samples from adjacent CUs can be considered unavailable for intra-frame prediction.

[0110] The basic unit for the output of the H.264 / AVC or HEVC encoder and the input of the H.264 / AVC or HEVC decoder, respectively, is the Network Abstraction Layer (NAL) unit. For transmission on a packet-oriented network or storage in a structured file, the NAL unit can be encapsulated into a packet or similar structure. The byte stream format has been specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide a frame structure. The byte stream format separates NAL units from each other by attaching a start code in front of each NAL unit. In order to avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm that adds emulation prevention bytes to the NAL unit payload if the start code would otherwise occur. In order to enable direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention can always be performed regardless of whether the byte stream format is used. The NAL unit can be defined as a syntax structure containing an indication of the type of data to be followed and bytes containing the data in the form of RBSP and interspersed with emulation prevention bytes as needed. A Raw Byte Sequence Payload (RBSP) may be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP is either empty or in the form of a data bit string containing a syntax element followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0111] NAL includes a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0112] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains a reserved bit, a six-bit NAL unit type indication, a three-bit nuh_temporal_id_plus1 indication for the time level (which may be required to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered as a time identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1-1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest time level. The value of temporal_id_plus1 is required to be non-zero in order to avoid start code emulation involving two NAL unit header bytes. The bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to the selected value and including all other VCL NAL units remains consistent. Therefore, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as an inter-frame prediction reference. A sublayer or temporal sublayer may be defined as a temporal scalable layer (or temporal layer TL) of a temporal scalable bitstream, which includes VCL NAL units and associated non-VCL NAL units with a specific value for the TemporalId variable. Pictures at a temporal sublayer with TemporalId equal to N may be predicted based on reference pictures at temporal sublayers with TemporalId less than or equal to N (unless further constrained by picture type, as explained below), and are not allowed or prohibited from being predicted based on reference pictures at temporal sublayers with TemporalId greater than N. nuh_layer_id may be understood as a scalable layer identifier.

[0113] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually coded slice NAL units. In HEVC, VCL NAL units contain syntax elements representing one or more CUs.

[0114] In HEVC, abbreviations for picture types may be defined as follows: trailing (TRAIL) picture, temporal sub-layer access (TSA), step-by-step temporal sub-layer access (STSA), random access decodable leading (RADL) picture, random access skip leading (RASL) picture, broken link access (BLA) picture, instantaneous decoding refresh (IDR) picture, clean random access (CRA) picture.

[0115] A random access point (RAP) picture (which may also be referred to as an intra random access point (IRAP) picture in an independent layer) contains only intra-coded slices. An IRAP picture belonging to a prediction layer may contain P, B, and I slices, may not use inter-frame prediction from other pictures in the same prediction layer, and may use inter-layer prediction from its direct reference layer. In the current version of HEVC, an IRAP picture may be a BLA picture, a CRA picture, or an IDR picture. The first picture in the bitstream containing the base layer is an IRAP picture in the base layer. If the necessary parameter set is available when it needs to be activated, the IRAP picture in the independent layer and all subsequent non-RASL pictures in the decoding order of the independent layer can be correctly decoded without performing the decoding process of any picture before the IRAP picture in the decoding order. When the necessary parameter set is available when it needs to be activated and when the decoding of each direct reference layer of the prediction layer has been initialized, the IRAP picture belonging to the prediction layer and all subsequent non-RASL pictures in the decoding order within the same prediction layer can be correctly decoded without performing the decoding process of any picture of the same prediction layer before the IRAP picture in the decoding order. There may be pictures in the bitstream that contain only intra-coded slices, which are non-IRAP pictures.

[0116] There are two picture types in HEVC that can be used to indicate temporal sublayer switching points, TSA and STAT picture types. If the temporal sublayer with TemporalId up to N has been decoded, until the TSA or STSA picture (excluded) and the TSA or STSA picture has a TemporalId equal to N+1, the TSA or STSA picture enables all subsequent pictures (in decoding order) with TemporalId equal to N+1 to be decoded. The TSA picture type can impose restrictions on the TSA picture itself and all pictures in the same sublayer after the TSA picture in decoding order. None of these pictures are allowed to use inter-frame prediction from any picture in the same sublayer before the TSA picture in decoding order. The TSA definition can further impose restrictions on pictures in higher sublayers after the TSA picture in decoding order. None of these pictures are allowed to refer to pictures before the TSA picture in decoding order if the picture belongs to the same or higher sublayer as the TSA picture. The TSA picture has a TemporalId greater than 0. STSA is similar to TSA pictures, but does not impose restrictions on pictures in higher sub-layers following the STSA picture in decoding order, thus enabling up-switching only to the sub-layer where the STSA picture resides.

[0117] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end-of-sequence NAL unit, an end-of-bitstream NAL unit, or a filler data NAL unit. Parameter sets may be required for the reconstruction of decoded pictures, while many other non-VCL NAL units are not necessary for the reconstruction of decoded sample values.

[0118] Parameters that remain unchanged through the encoded video sequence may be included in a sequence parameter set. In addition to parameters that may be required for the decoding process, the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that may be referenced by one or more picture parameter set RBSPs or one or more SEI NAL units containing a buffering period SEI message. The picture parameter set contains such parameters that may remain unchanged in several coded pictures. The picture parameter set RBSP may include parameters that may be referenced by the coded slice NAL units of one or more coded pictures.

[0119] In HEVC, a video parameter set (VPS) can be defined as a syntax structure containing syntax elements that apply to zero or more complete coded video sequences, as determined by the contents of syntax elements found in the SPS, which are referenced by syntax elements found in the PPS, which are referenced by syntax elements found in each slice segment header.

[0120] A video parameter set RBSP may include parameters that may be referenced by one or more sequence parameter set RBSPs.

[0121] The relationship and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS) and picture parameter sets (PPS) can be described as follows. The VPS resides one layer above the SPS in the parameter set hierarchy and resides in the context of scalability and / or 3D video. The VPS may include parameters common to all slices across all (scalability or view) layers in a complete coded video sequence. The SPS includes parameters common to all slices in a specific (scalability or view) layer in a complete coded video sequence, and may be shared by multiple (scalability or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalability or view layer in an access unit), and may be shared by all slices in multiple layer representations.

[0122] The VPS may provide information about dependencies of layers in the bitstream and many other information applicable to all slices across all (scalability or view) layers in a complete coded video sequence. The VPS may be considered to consist of two parts, a base VPS and a VPS extension, where the VPS extension may optionally be present.

[0123] Additionally or alternatively, out-of-band transmission, signaling, or storage may be used for purposes other than tolerance of transmission errors, such as ease of access or session negotiation. For example, sample entries for a track in a file conforming to the ISO base media file format may include parameter sets, while the coded data in the bitstream is stored elsewhere in the file or in another file. The phrases "along the bitstream (e.g., indicated along the bitstream)" or "along the coding units of the bitstream (e.g., indicated along the coding blocks)" may be used in the claims and described embodiments to refer to out-of-band transmission, signaling, or storage in a manner in which the out-of-band data is associated with the bitstream or coding units, respectively. The phrases "decoding along the bitstream or along the coding units of the bitstream, etc." may refer to decoding the referred out-of-band data associated with the bitstream or coding units, respectively (which may be obtained from out-of-band transmission, signaling, or storage). For example, the phrase "along the bitstream" may be used when a bitstream is contained in a file (such as a file conforming to the ISO base media file format), and certain file metadata is stored in the file in a manner that associates the metadata with the bitstream, such as in a sample entry for a track containing the bitstream, a sample group for a track containing the bitstream, or a box in a timed metadata track associated with a track containing the bitstream.

[0124] The SEI NAL unit may contain one or more SEI messages that are not required for decoding of the output picture, but may assist related processes such as picture output timing, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264 / AVC and HEVC, and the user data SEI message enables organizations and companies to specify SEI messages for their own use. H.264 / AVC and HEVC contain syntax and semantics for the specified SEI messages, but do not define the process for processing the messages at the receiver. Therefore, the encoder needs to follow the H.264 / AVC standard or the HEVC standard when creating SEI messages, and decoders that conform to the H.264 / AVC standard or the HEVC standard, respectively, do not need to process SEI messages for output order consistency. One of the reasons for including the syntax and semantics of SEI messages in H.264 / AVC and HEVC is to allow different system specifications to interpret supplementary information identically and thus interoperate. It is contemplated that system specifications may require the use of specific SEI messages both at the encoding end and at the decoding end, and furthermore, may specify procedures for handling specific SEI messages in the receiving end.

[0125] In HEVC, there are two types of SEI NAL units, suffix SEI NAL units and prefix SEI NAL units, which have different nal_unit_type values ​​from each other. The (one or more) SEI messages contained in the suffix SEI NAL unit are associated with the VCL NAL units that precede the suffix SEI NAL unit in the decoding order. The (one or more) SEI messages contained in the prefix SEI NAL unit are associated with the VCL NAL units that follow the prefix SEI NAL unit in the decoding order.

[0126] A coded picture is an encoded representation of a picture.

[0127] In HEVC, a coded picture can be defined as a coded representation of a picture that contains all coding tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any specific value of nuh_layer_id. In addition to VCL NAL units containing coded pictures, access units may also contain non-VCL NAL units. The specified classification rules may, for example, associate pictures with the same output time or picture output count value into the same access unit.

[0128] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or a byte stream, which forms a representation of a coded picture and associated data forming one or more coded video sequences. A first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which can be called an end-of-bitstream (EOB) NAL unit and is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have nuh_layer_id equal to 0.

[0129] In H.264 / AVC, a coded video sequence is defined as a sequence of consecutive access units in decoding order from an IDR access unit (inclusive) to the next IDR access unit (exclusive) or to the end of the bitstream, whichever comes first.

[0130] In HEVC, a coded video sequence (CVS) may be defined, for example, as a sequence of access units that includes, in decoding order, an IRAP access unit with NoRaslOutputFlag equal to 1, followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1, including all subsequent access units up to but not including any subsequent access unit that is an IRAP access unit with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit for which the base layer picture is an IRAP picture. For each IDR picture, the value of NoRaslOutputFlag is equal to 1, each BLA picture and each IRAP picture (which is the first picture in that particular layer in the bitstream in decoding order) is the first IRAP picture in decoding order after the sequence-ending NAL unit with the same value of nuh_layer_id. There may be a method for providing the value of HandleCraAsBlaFlag to a decoder from an external entity (e.g., a player or receiver) that can control the decoder. HandleCraAsBlaFlag may be set to 1, for example, by a player that seeks to a new position in the bitstream or tunes to a broadcast and starts decoding, then starts decoding from a CRA picture. When HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is processed and decoded as if it were a BLA picture.

[0131] In HEVC, additionally or alternatively (to the above description), a coded video sequence may be designated as ending when a specific NAL unit (which may be referred to as an end of sequence (EOS) NAL unit) appears in the bitstream and has nuh_layer_id equal to 0.

[0132] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous picture is decoded. An open GOP is a group of pictures where, when decoding starts from the initial intra picture of an open GOP, pictures before the initial intra picture in output order may not be decoded correctly. In other words, pictures in an open GOP can refer to pictures belonging to the previous GOP (in inter prediction). The HEVC decoder can identify the intra picture that starts an open GOP because a specific NAL unit type, a CRA NAL unit type, can be used for its coded slice. A closed GOP is a group of pictures where, when decoding starts from the initial intra picture of a closed GOP, all pictures can be decoded correctly. In other words, no picture in a closed GOP refers to any picture in a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start from an IDR picture. In HEVC, a closed GOP can also start from a BLA_W_RADL or BLA_N_LP picture. Due to the greater flexibility in the selection of reference pictures, the open GOP coding structure is potentially more efficient in compression than the closed GOP coding structure.

[0133] The decoded picture buffer (DPB) can be used in the encoder and / or decoder. There are two reasons for buffering decoded pictures, for reference in inter-frame prediction and for reordering decoded pictures to output order. Since H.264 / AVC and HEVC provide great flexibility for reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering can waste memory resources. Therefore, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. When a decoded picture is no longer used as a reference and is not needed for output, it can be removed from the DPB.

[0134] In many coding modes of H.264 / AVC and HEVC, reference pictures used for inter prediction are indicated by an index to a reference picture list. The index can be encoded with variable length coding, which generally results in smaller indices having shorter values ​​for corresponding syntax elements. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bidirectionally predicted (B) slice, and one reference picture list (reference picture list 0) is formed for each intra-coded (P) slice.

[0135] Many coding standards (including H.264 / AVC and HEVC) may have a decoding process to derive a reference picture index to a reference picture list, which may be used to indicate which of multiple reference pictures is used for inter prediction of a particular block. The reference picture index may be encoded into the bitstream by the encoder in some inter coding modes, or it may be derived (by the encoder and decoder) using neighboring blocks, for example, in some other inter coding modes.

[0136] For a single prediction unit, multiple candidate motion vectors can be derived. For example, motion vector prediction HEVC includes two motion vector prediction schemes, namely advanced motion vector prediction (AMVP) and merge mode. In AMVP or merge mode, a motion vector candidate list is derived for the PU. There are two types of candidates: spatial candidates and temporal candidates, where temporal candidates can also be referred to as TMVP candidates.

[0137] The candidate list derivation can be performed, for example, as follows, however it should be understood that there may be other possibilities for the candidate list derivation. If the occupancy of the candidate list is not at a maximum value, the spatial candidates are first included in the candidate list if these spatial candidates are available and do not already exist in the candidate list. Afterwards, if the occupancy of the candidate list is not yet at a maximum value, the temporal candidates are included in the candidate list. If the number of candidates still does not reach the maximum allowed number, a combined bidirectional prediction candidate (for B slices) and a zero motion vector are added. After the candidate list has been constructed, the encoder decides the final motion information from the candidates, for example based on a rate-distortion optimization (RDO) decision, and encodes the index of the selected candidate into the bitstream. Similarly, the decoder decodes the index of the selected candidate from the bitstream, constructs the candidate list, and uses the decoded index to select a motion vector predictor from the candidate list.

[0138] In HEVC, AMVP and merge mode can be characterized as follows. In AMVP, the encoder indicates whether to use unidirectional or bidirectional prediction and which reference pictures to use, and encodes the motion vector difference. In merge mode, only the candidates selected from the candidate list are encoded into the bitstream, which indicates that the current prediction unit has the same motion information as the indicated predictor. Therefore, merge mode creates regions consisting of adjacent prediction blocks sharing the same motion information, which is signaled only once for each region.

[0139] An example of the operation of advanced motion vector prediction is provided below, and other similar implementations of advanced motion vector prediction are possible, such as having different candidate position sets and candidate positions with candidate position sets. It is also necessary to understand that other prediction modes (such as merge mode) can operate similarly. Two spatial motion vector predictors (MVPs) can be derived, and a temporal motion vector predictor (TMVP) can be derived. They can be selected from the following positions: three spatial motion vector predictor candidate positions (B0, B1, B2) located above the current prediction block and two on the left (A0, A1). The first motion vector predictor (B0, B1, B2) or (A0, A1) available in a predefined order for each candidate position set (e.g., residing in the same strip, being intra-coded, etc.) can be selected to indicate the prediction direction (up or left) in the motion vector competition. The reference index for the temporal motion vector predictor can be indicated by the encoder in the stripe header (e.g., as a collocated_ref_idx syntax element). The first motion vector predictor that is available (e.g., inter-coded) in a predefined order of potential temporal candidate positions (e.g., in order (C0, C1)) can be selected as a source for the temporal motion vector predictor. The motion vector obtained from the first available candidate position in the co-located picture can be scaled according to the ratio of the picture order count difference of the reference picture, the co-located picture, and the current picture of the temporal motion vector predictor. Moreover, a redundancy check can be performed among the candidates to remove identical candidates, which can result in zero motion vectors being included in the candidate list. The motion vector predictor can be indicated in the bitstream, for example, by indicating the direction of the spatial motion vector predictor (up or left) or the selection of the temporal motion vector predictor candidate. The co-located picture may also be referred to as a collocated picture, a source for motion vector prediction, or a source picture for motion vector prediction.

[0140] The motion parameter type or motion information may include but is not limited to one or more of the following types:

[0141] - indication of the prediction type (e.g. intra prediction, unidirectional prediction, bidirectional prediction) and / or number of reference pictures;

[0142] - an indication of a prediction direction, such as inter-frame (also called temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP), and inter-component prediction (which may be indicated per reference picture and / or per prediction type, and in some embodiments, inter-view prediction and view synthesis prediction may be jointly considered as one prediction direction), and / or

[0143] - an indication of a reference picture type, such as a short-term reference picture and / or a long-term reference picture and / or an inter-layer reference picture (which may be indicated, for example, per reference picture);

[0144] - a reference index to a reference picture list and / or any other identifier of a reference picture (which may be indicated, for example, per reference picture, and its type may depend on the prediction direction and / or the reference picture type, and it may be accompanied by other relevant information, such as the reference picture list to which the reference index applies or similar information);

[0145] - horizontal motion vector component (which may be indicated, for example, per prediction block or per reference index, etc.);

[0146] - vertical motion vector component (which may be indicated, for example, by prediction block or by reference index, etc.);

[0147] - one or more parameters, such as a picture order count difference and / or a relative camera separation between a picture containing or associated with a motion parameter and its reference pictures, which may be used for scaling of a horizontal motion vector component and / or a vertical motion vector component in one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference picture or per reference index, etc.);

[0148] - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left sample of the block in units of luma samples;

[0149] - The extent (eg width and height) of the block to which the motion parameters and / or motion information apply.

[0150] Typically, motion vector prediction mechanisms, such as those presented above as examples, may include the prediction or inheritance of certain predefined or indicated motion parameters.

[0151] A motion field associated with a picture may be considered to include a set of motion information generated for each coded block of the picture. For example, the motion field may be accessed by the coordinates of the block. The motion field may be used, for example, in TMVP or any other motion prediction mechanism in which a source or reference other than the currently (de)coded picture is used for prediction.

[0152] Different spatial granularities or units may be applied to represent and / or store motion fields. For example, a regular grid of spatial units may be used. For example, a picture may be divided into rectangular blocks of a specific size (except for blocks at the edges of the picture, such as at the right and bottom edges). For example, the size of the spatial unit may be equal to the minimum size that different motions may be indicated by the encoder in the bitstream, such as a 4×4 block in units of luminance samples. For example, a so-called compressed motion field may be used, in which a spatial unit may be equal to a predefined or indicated size, such as a 16×16 block in units of luminance samples, which may be larger than the minimum size for indicating different motions. For example, the HEVC encoder and / or decoder may be implemented in a manner that performs motion data storage reduction (MDSR) or motion field compression for each decoded motion field (before using the motion field for any prediction between pictures). In the HEVC implementation, MDSR may reduce the granularity of motion data to 16×16 blocks in units of luminance samples by keeping the upper left corner sample applicable to the 16×16 block in the motion compression motion field. The encoder may encode (one or more) indications related to the spatial units of the compressed motion field, for example as one or more syntax elements and / or syntax element values ​​in a sequence level syntax structure (such as a video parameter set or a sequence parameter set). In some (de)coding methods and / or devices, the motion field may be represented and / or stored according to block partitions for motion prediction (e.g., prediction units according to the HEVC standard). In some (de)coding methods and / or devices, a combination of regular grid and block partitioning may be applied so that motion associated with partitions larger than a predefined or indicated spatial unit size is represented and / or stored in association with those partitions, while motion associated with partitions smaller than or aligned with a predefined or indicated spatial unit size or grid is represented and / or stored for predefined or indicated units.

[0153] Scalable video coding may refer to a coding structure in which one bitstream may contain multiple representations of the content (e.g., at different bitrates, resolutions, or frame rates). In these cases, the receiver may extract the desired representation depending on its characteristics (e.g., the resolution that best matches the display device). Alternatively, a server or network element may extract parts of the bitstream to be sent to the receiver depending on, for example, the network characteristics or processing power of the receiver. A meaningful decoded representation may be generated by decoding only certain parts of the scalable bitstream. A scalable bitstream typically includes a "base layer" that provides the lowest quality available video and one or more enhancement layers that enhance the video quality when received and decoded with the lower layers. In order to improve the coding efficiency for the enhancement layer, the coded representation of the layer typically depends on the lower layer. For example, the motion and pattern information of the enhancement layer can be predicted from the lower layer. Similarly, the pixel data of the lower layer can be used to create a prediction for the enhancement layer.

[0154] In some scalable video coding schemes, a video signal may be encoded as a base layer and one or more enhancement layers. The enhancement layer may enhance, for example, temporal resolution (i.e., frame rate), spatial resolution, or simply the quality of the video content represented by another layer or a portion thereof. Each layer together with all its dependent layers is a representation of the video signal, for example, at a specific spatial resolution, temporal resolution, and quality level. In this document, a scalable layer together with all its associated layers is referred to as a "scalable layer representation." The portion of the scalable bitstream corresponding to the scalable layer representation may be extracted and decoded to produce a representation of the original signal at a specific fidelity.

[0155] Scalability modes or scalability dimensions may include, but are not limited to, the following:

[0156] - Quality scalability: Base layer pictures are encoded with lower quality than enhancement layer pictures, which can be achieved, for example, by using larger quantization parameter values ​​(i.e., larger quantization step sizes for transform coefficient quantization) in the base layer than in the enhancement layer. Quality scalability can be further classified as fine granularity or fine granularity scalability (FGS), medium granularity or medium granularity scalability (MGS), and / or coarse granularity or coarse granularity scalability (CGS), as described below.

[0157] - Spatial scalability: Base layer pictures are coded at a lower resolution (ie, with fewer samples) than enhancement layer pictures. Spatial scalability and quality scalability (particularly its coarse-grained scalability type) can sometimes be considered the same scalability type.

[0158] - Bit depth scalability: Base layer pictures are encoded at a lower bit depth (eg 8 bits) than enhancement layer pictures (eg 10 or 12 bits).

[0159] - Dynamic range scalability: Scalable layers represent different dynamic ranges and / or images obtained using different tone mapping functions and / or different optical transfer functions.

[0160] - Chroma format scalability: Base layer pictures provide lower spatial resolution in chroma sample arrays (eg, encoded in 4:2:0 chroma format) than enhancement layer pictures (eg, 4:4:4 format).

[0161] - Color gamut scalability: The enhancement layer pictures have a richer / wider color representation range than the base layer pictures; for example, the enhancement layer may have a UHDTV (ITU-R BT.2020) color gamut, while the base layer may have an ITU-R BT.709 color gamut.

[0162] - View scalability, which may also be referred to as multi-view coding. The base layer represents the first view, while the enhancement layer represents the second view. A view may be defined as a sequence of pictures representing one camera or viewpoint. It may be thought of as stereo or dual-view video, where one video sequence or view is presented to the left eye, and parallel views are presented to the right eye.

[0163] - Depth scalability, which may also be referred to as depth enhancement coding. One or some layers of the bitstream may represent texture view(s), while other one or more layers may represent depth view(s).

[0164] - Region of Interest Scalability (as described below).

[0165] - Interlaced to progressive scalability (also known as field to frame scalability): The coded interlaced source content material of the base layer is enhanced with an enhancement layer to represent the progressive source content. The coded interlaced source content in the base layer may include coded fields, coded frames representing pairs of fields, or a mixture thereof. In interlaced to progressive scalability, a base layer picture may be resampled so that it becomes a suitable reference picture for one or more enhancement layer pictures.

[0166] - Hybrid codec scalability (also known as coding standard scalability): In hybrid codec scalability, the bitstream syntax, semantics, and decoding processes of the base layer and enhancement layer are specified in different video coding standards. Therefore, the base layer pictures are encoded according to a different coding standard or format than the enhancement layer pictures. For example, the base layer can be encoded with H.264 / AVC and the enhancement layer can be encoded with the HEVC multi-layer extension.

[0167] It should be understood that many scalability types can be combined and applied together. For example, color gamut scalability and bit depth scalability can be combined.

[0168] The term "layer" may be used in the context of any type of scalability including view scalability and depth enhancement. An enhancement layer may refer to any type of enhancement, such as SNR, spatial, multi-view, depth, bit depth, chroma format, and / or color gamut enhancement. A base layer may refer to any type of base video sequence, such as a base view, a base layer for SNR / spatial scalability, or a texture base view for depth enhanced video coding.

[0169] Some scalable video coding schemes may require IRAP pictures to be aligned across layers in such a way that all pictures in an access unit are IRAP pictures or no pictures in an access unit are IRAP pictures. Other scalable video coding schemes (such as multi-layer extensions of HEVC) may allow unaligned IRAP pictures, i.e., one or more pictures in an access unit are IRAP pictures, while one or more other pictures in the access unit are not IRAP pictures. Scalable bitstreams with IRAP pictures or the like that are not aligned across layers may be used, for example, to provide more frequent IRAP pictures in a base layer, where they may have a smaller coding size due to, for example, a smaller spatial resolution. A process or mechanism for starting decoding layer by layer may be included in a video decoding scheme. Thus, the decoder may start decoding of the bitstream when the base layer contains an IRAP picture, and gradually start decoding other layers when other layers contain IRAP pictures. In other words, in a mechanism or process for starting decoding layer by layer, the decoder gradually increases the number of decoding layers (wherein the layer may represent an enhancement to spatial resolution, quality level, view, additional component (such as depth), or combination) because subsequent pictures from additional enhancement layers are decoded during the decoding process. A gradual increase in the number of decoded layers may be perceived, for example, as a gradual improvement in picture quality (in the case of quality and spatial scalability).

[0170] A transmitter, a gateway, a client or another entity may select the transmission layer and / or sublayer of a scalable video bitstream. The term "layer extraction", "layer extraction" or "layer down switching" may refer to sending fewer layers than the layers available in the bitstream received by the transmitter, a gateway, a client or another entity. Layer up switching may refer to sending (one or more) additional layers compared to those layers sent by the transmitter, a gateway, a client or another entity before the layer up switching, i.e. restarting the transmission of one or more layers whose transmission was stopped earlier in the layer down switching. Similar to layer down switching and / or up switching, a transmitter, a gateway, a client or another entity may perform down and / or up switching of a temporal sublayer. A transmitter, a gateway, a client or another entity may also perform layer and sublayer down switching and / or up switching. Layer and sublayer down switching and / or up switching may be performed in the same access unit, etc. (i.e., substantially at the same time) or may be performed in different access units, etc. (i.e., substantially at different times).

[0171] Scalability can be enabled in two basic ways. By introducing new coding modes for performing prediction of pixel values ​​or syntax from lower layers of scalable representations, or by placing lower layer pictures into a reference picture buffer (e.g., decoded picture buffer, DPB) of a higher layer. The first approach may be more flexible and therefore may provide better coding efficiency in most cases. However, the second approach of reference frame-based scalability can be effectively implemented with minimal changes to a single-layer codec while still achieving most of the available coding efficiency gains. Basically, a reference frame-based scalability codec can be implemented by using the same hardware or software implementation for all layers, with DPB management handled only externally.

[0172] A scalable video encoder for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability can be implemented as follows. For the base layer, a non-scalable video encoder and decoder can be used. The reconstructed / decoded picture of the base layer is included in the reference picture buffer and / or reference picture list for the enhancement layer. In the case of spatial scalability, the reconstructed / decoded base layer picture can be upsampled before it is inserted into the reference picture list for the enhancement layer picture. Similar to the decoded reference picture of the enhancement layer, the base layer decoded picture can be inserted into (one or more) reference picture lists for encoding / decoding of the enhancement layer picture. Therefore, the encoder can select the base layer reference picture as an inter-frame prediction reference and indicate its use with a reference picture index in the encoded bitstream. The decoder decodes the base layer picture from the bitstream (e.g., from the reference picture index) to be used as an inter-frame prediction reference for the enhancement layer. When the decoded base layer picture is used as a prediction reference for the enhancement layer, it is called an inter-layer reference picture.

[0173] Although the previous paragraphs describe a scalable video codec with two scalability layers (an enhancement layer and a base layer), it should be understood that the description can be generalized to any two layers in a scalability hierarchy with more than two layers. In this case, the second enhancement layer can rely on the first enhancement layer during encoding and / or decoding, and therefore, the first enhancement layer can be considered as a base layer for encoding and / or decoding of the second enhancement layer. In addition, it should be understood that there may be inter-layer reference pictures from more than one layer in the reference picture buffer or reference picture list of the enhancement layer, and each of these inter-layer reference pictures can be considered to reside in the base layer or reference layer for the enhancement layer being encoded and / or decoded. In addition, it should be understood that other types of inter-layer processing other than upsampling of the reference layer pictures can be performed instead or in addition. For example, the bit depth of the samples of the reference layer picture can be converted to the bit depth of the enhancement layer, and / or the sample values ​​can undergo a mapping from the color space of the reference layer to the color space of the enhancement layer.

[0174] Inter-layer prediction can be defined as prediction in a way that relies on data elements (e.g., sample values ​​or motion vectors) from a reference picture of a layer different from the layer of the current picture (being encoded or decoded). There are many types of inter-layer prediction and can be applied in a scalable video encoder / decoder. The available inter-layer prediction types may, for example, depend on the coding profile according to which the bitstream or a particular layer within the bitstream is being encoded, or the coding profile to which the bitstream or a particular layer within the bitstream is indicated to conform when decoding. Alternatively or additionally, the available inter-layer prediction types may depend on the type of scalability or type of scalable codec or video coding standard revision being used (e.g., SHVC, MV-HEVC, or 3D-HEVC).

[0175] A direct reference layer may be defined as a layer that can be used for inter-layer prediction of another layer, where the layer is a direct reference layer for the other layer. A direct prediction layer may be defined as a layer for which the other layer is a direct reference layer. An indirect reference layer may be defined as a layer that is not a direct reference layer of a second layer but a direct reference layer of a third layer, where the third layer is a direct reference layer or an indirect reference layer of a direct reference layer of a second layer, where the layer is an indirect reference layer for the second layer. An indirect prediction layer may be defined as a layer for which the other layer is an indirect reference layer. An independent layer may be defined as a layer that does not have a direct reference layer. In other words, an independent layer is not predicted using inter-layer prediction. A non-base layer may be defined as any other layer except a base layer, and a base layer may be defined as the lowest layer in a bitstream. An independent non-base layer may be defined as a layer that is both an independent layer and a non-base layer.

[0176] A neural network (NN) is a computational graph comprising several computational layers. Each layer comprises one or more units, each of which performs a basic computation. Units are connected to one or more other units, and the connections may have associated weights. Weights may be used to scale signals passing through the associated connections. Weights are typically learnable parameters, i.e., values ​​that can be learned from training data. There may be other learnable parameters, such as parameters of a batch-normalization layer.

[0177] Two example architectures for neural networks are feedforward architectures and recursive architectures. A feedforward neural network is such that there is no feedback loop: each layer takes input from one or more of the previous layers and provides its output as input to one or more of the subsequent layers. Also, units within a particular layer take input from units in one or more of the previous layers and provide output to one or more of the subsequent layers.

[0178] The initial layers (those close to the input data) extract semantic low-level features, such as edges and textures in an image, and the intermediate and final layers extract more high-level features. After the feature extraction layers, there may be one or more layers that perform specific tasks such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc. In a recurrent neural network, there is a feedback loop so that the network becomes stateful, i.e., it is able to remember information or state.

[0179] Neural networks are being used in an ever-increasing number of applications for many different types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0180] The property of neural networks (and other machine learning tools) is that they are able to learn features from input data, either in a supervised or unsupervised manner. This learning can be the result of a training algorithm, or a meta-level neural network that provides the training signal.

[0181] Typically, a training algorithm may include changing some characteristics of a neural network so that its output is as close to the desired output as possible. For example, in the case of classification of objects in an image, the output of the neural network may be used to derive a class or category index indicating the class or category to which the object in the input image belongs. Training may include minimizing or reducing the error (also called loss) of the output. Examples of losses are mean squared error, cross entropy, etc. In deep learning techniques, training may be an iterative process, where at each iteration the algorithm modifies the weights of the neural network to make incremental improvements to the output of the network, i.e., to gradually reduce the loss.

[0182] In the context of training neural networks, an "epoch" refers to the time that the neural network is used on all of the training data, and it may include multiple iterations. An "iteration" refers to the time that the neural network is used on a subset of the entire training data, often referred to as a batch or mini-batch of the data. Typically, an update to the weights of the neural network is performed at each iteration. In particular, a batch of data is input to the network, output is obtained from the network, loss is calculated based on the output, the gradient of the loss with respect to the weights is calculated, and a weight update is determined based on the calculated gradient.

[0183] In this document, the terms "model", "neural network", "neural net" and "network" are used interchangeably, and the weights of a neural network are sometimes referred to as learnable parameters or simply parameters.

[0184] Training a neural network is an optimization process, but the final goal is different from the typical optimization goal. In normal optimization, the goal is simply to minimize the function. In machine learning, the goal of the optimization or training process is to make the model learn the characteristics of the data distribution from a limited training data set. In other words, the goal is to learn to use a limited training data set in order to learn to generalize to previously undiscovered data, that is, data that has not been used to train the model. This is often referred to as generalization. In practice, the data can be divided into at least two sets, a training set and a validation set. The training set is used to train the network, that is, to modify its learnable parameters in order to minimize loss. The validation set can be used to check the performance of the network on data that has not been used to minimize loss, as an indication of the final performance of the model. In particular, the errors on the training set and the validation set are monitored during the training process to understand the following:

[0185] - If the network is fully learning - in this case the training set error should decrease, otherwise the model is in the underfitting regime.

[0186] -If the network is learning to generalize -In this case, the validation set error also needs to decrease and needs to not be much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, then the model is in an overfitting regime. This means that the model has just memorized the characteristics of the training set and performs well only on that set, but does not perform well on sets that were not used to tune its parameters.

[0187] Returning to the hybrid video compression described above, it has been described how intra (i.e. spatial) and inter (e.g. temporal) prediction processes are performed not only at the decoder but also in the encoder (more specifically in the "decoding loop" of the encoder) in order to calculate the residual. Decoding is performed in a specific scan order, for example, the coding units can be encoded in a raster scan order within the coding tree units, and the coding tree units can be encoded in a raster scan order within the blocks. Therefore, for each block to be predicted, the already reconstructed pixels may be those above and to the left of the block. Of all the already reconstructed pixels above and to the left of the block to be predicted, only some (such as 2 or 4) rows and columns can be used for prediction.

[0188] Thus, intra prediction may involve using the pixels of the rows and columns above and to the left of the block to be predicted to deduce how the block might look. Blocks may be relatively small, e.g. 8 by 8 pixels, and thus the intra prediction process may be reduced to predicting small image patterns, such as edges or texture details or even constant / homogeneous regions. Figure 5An example of intra prediction input including the some rows and columns of pixels above and to the left of the area of ​​the block to be predicted and output for the block is shown.

[0189] The intra prediction process is usually performed using an algorithmic approach (i.e., without using a learning approach). Intra prediction usually involves predefined prediction modes, where each prediction mode refers to a different image mode, such as homogeneous content (the prediction area has the same constant value, which can be derived, for example, as the average sample value of selected samples of neighboring blocks or as the pixels already reconstructed to the left and top), deriving a smoothly varying prediction block (e.g., the planar mode of H.264 / AVC or HEVC), or a directional mode. The direction of the prediction mode used for intra prediction, i.e., the prediction direction to be applied in a specific prediction mode, can be, for example, a vertical, horizontal, or selected diagonal mode corresponding to different directional prediction angles. Directional prediction modes can be considered to correspond to edges with different angles / orientations. During encoding, all or several prediction modes are tried, and the best prediction mode is selected based on, for example, the mean square error (MSE) between the predicted block and the true block. The prediction mode is then signaled to the decoder, for example by including it in the residual bitstream of the block.

[0190] The goal of the intra prediction process is to obtain the minimum possible prediction error or residual, thus requiring fewer bits to encode the intra frames which consumes a large portion of the bitstream of the entire video. However, the algorithmic approaches described above are limited to this ultimate research and development which has saturated in terms of prediction accuracy. Therefore, further development can be achieved using learning methods.

[0191] An enhanced method for using a neural network for the intra prediction process is now introduced.

[0192] Figure 6The method depicted in the flowchart of includes: obtaining (600) a configuration of at least one neural network including a plurality of initial intra-prediction mode-agnostic layers and one or more intra-prediction mode-specific layers, the one or more intra-prediction mode-specific layers corresponding to different intra-prediction modes; obtaining (602) at least one input video frame including a plurality of blocks; determining (604) to encode the one or more blocks using intra-prediction; determining (606) an intra-prediction mode for each of the one or more blocks; grouping (608) blocks having the same intra-prediction mode into groups, each group being assigned a computational path among the plurality of intra-prediction mode-agnostic layers and the one or more intra-prediction mode-specific layers; training (610) the plurality of intra-prediction mode-agnostic layers and / or the one or more intra-prediction mode-specific layers of the neural network based on a training loss between an output of the neural network associated with the block group and an annotated block, wherein the annotated block is a block of the input video frame or a reconstructed block; and encoding the block using the computational path assigned to the intra-prediction mode for the block (612).

[0193] According to an embodiment, the method further comprises: in response to the configuration of the at least one neural network comprising a pre-trained neural network, using blocks of the input video frame as the labeled blocks. Thus, when the neural network is pre-defined, or when the neural network is pre-trained by the encoder and sent to the decoder, the blocks of the input video frame can be used as the labeled blocks used in the training phase (610).

[0194] Since the intra-frame prediction neural networks in the encoder and decoder are preferably aligned (i.e., operated with the same parameters), the intra-frame prediction neural networks in the encoder and decoder can be trained in exactly the same way or at least substantially in the same way. According to an embodiment, the encoder indicates information indicating which blocks will be used for training in or along the bitstream. Here, the encoder can encode the training blocks (i.e., groups of blocks using the same intra-frame prediction mode) with higher quality. In one embodiment, the decoded or reconstructed blocks corresponding to the encoded training blocks are used by the decoder as annotation data to calculate the training loss. Similarly, in an embodiment, the encoder uses the decoded or reconstructed blocks corresponding to these encoded training blocks instead of the original blocks as annotations.

[0195] exist Figure 7The corresponding decoding method depicted in the flowchart of includes: obtaining (700) a configuration of at least one neural network including multiple intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers corresponding to different intra-prediction modes; obtaining (702) a bitstream including an encoded video frame, wherein the encoded video frame includes a plurality of blocks; decoding (704) from or along the bitstream information indicating a block group defined as a training block; decoding (706) the training block into a reconstructed block; training (708) the multiple intra-prediction mode agnostic layers and / or the one or more intra-prediction mode specific layers of the neural network based on a training loss between an output of the neural network related to the block group and the reconstructed block; and decoding (710) the encoded block using a calculation path of the intra-prediction mode assigned to the encoded block.

[0196] Thus, the method involves video compression using a neural network for an intra prediction process as a component of a video codec. Thus, the learning capabilities of the neural network are exploited to perform the intra prediction process in video compression, wherein the prediction mode is exploited in order to adapt the neural network to the specific content in the most efficient manner. In other words, the neural network is adapted to the prediction mode information at the inference stage.

[0197] It should be understood that although the figures and embodiments are described with reference to blocks or pictures, the embodiments may be applied on the basis of any spatial unit ranging from blocks to pictures. For example, embodiments may be similarly implemented for slices. Embodiments may also be applied to multiple intra frames. For example, training blocks for a particular intra prediction type may be collected from multiple intra frames (e.g., from multiple consecutive intra frames that may belong to the same scene).

[0198] The neural network can be delivered to the encoder and decoder, for example, in one of the following ways:

[0199] - The neural network may be predefined, for example, in a coding standard and thus known to both the encoder and the decoder.

[0200] - The neural network can be pre-trained. The pre-training can be performed by an entity external to the encoder and decoder, and delivered to both the encoder and decoder before encoding or decoding the bitstream.

[0201] - Pre-training may be performed by the encoder. The decoder may have means for obtaining a pre-trained network. For example, the network may be included in or along the bitstream, and the decoder may decode the neural network from the bitstream. For example, a specific parameter set (e.g., referred to as a neural network parameter set) may be specified for delivering the neural network in or along the bitstream. The use of a specific neural network parameter set may be indicated, for example, at the sequence, picture, or slice level, for example in a sequence parameter set, a picture parameter set, or a slice header, respectively.

[0202] When the neural network is delivered to the encoder and / or decoder, embodiments are not limited to any particular neural network representation syntax. The neural network type (e.g., fully convolutional neural network), basic parameters (e.g., number of layers), and parameters or weights of the neural network can be included in the neural network representation.

[0203] Embodiments are not limited to any particular type of neural network.

[0204] Described in more detail below Figure 6 and Figure 7 The steps of the method and their related embodiments are shown.

[0205] A plurality of prediction mode agnostic layers and a plurality of prediction mode related layers corresponding to different intra prediction modes Pre-trained configuration of a neural network with layers (600, 700)

[0206] Herein, the term "prediction mode agnostic layer" refers to a set of layers of an intra prediction neural network including a data block that may belong to any intra prediction mode. The term "prediction mode dependent layer" refers to a set of layers of an intra prediction neural network that processes blocks belonging to a specific intra prediction mode. For example, in response to an encoder or decoder determining to use a first intra prediction mode for a block group, the intra prediction neural network may use (one or more) intra prediction mode agnostic layers and a first intra prediction mode dependent layer set determined based on the first intra prediction mode. In response to determining to use a second intra prediction mode, the intra prediction neural network may use an intra prediction mode agnostic layer and a second intra prediction mode dependent layer set determined based on the second intra prediction mode. In one embodiment, the intra prediction mode agnostic layer set includes one or more initial layers of the network, for example, layer 1 to layer N at the input of the network, where N is the number of initial layers. In one embodiment, the intra prediction mode specific layer set includes one or more final layers, for example, layer K-M+1 to layer K at the output of the neural network, where MK is the total number of layers for a specific prediction mode. The partitioning into an intra-prediction mode agnostic part and an intra-prediction mode specific part may not always be layer-based, many different levels of partitioning may be applied, for example in some embodiments the partitioning may be at the filter level or even at the level of a single neuron. In general, there may be an intra-prediction mode agnostic part of the neural network and one or more intra-prediction mode specific parts of the network.

[0207] In the intra prediction process, the video frame can be divided into a group of blocks. Each block can be encoded separately. The encoder can determine or be configured to encode a specific frame as an intra-coded frame. For example, the first frame in the bitstream is usually an intra-coded frame. Intra-coded frames can be used to facilitate random access to the bitstream, so they can be encoded and inserted into the bitstream in a periodic manner. In addition, intra-coded frames can be encoded for scene cut positions because inter-frame prediction cannot be beneficial between scenes of different contents. In some cases, the encoder can determine or be configured to perform gradual decoding refresh, in which the picture content is refreshed on multiple pictures in the following manner: the refresh area grows in the decoding order of the picture, and the coding block in the refresh area is encoded with intra-frame mode or predicted from the refresh area of ​​the picture earlier in the decoding order among multiple pictures. For inter-coded frames, the video codec decides whether the block is encoded in intra-frame mode or in inter-frame mode, for example by first checking whether the inter-frame prediction meets the bit rate and peak signal noise (PSNR) requirements imposed by the user on the video to be compressed. If not, the block is encoded in intra mode, i.e. using only pixels in the current frame that have been decoded and reconstructed.Due to the typical raster scan order in decoding, the decoded and reconstructed pixels are usually located in the top row and left column relative to the block to be predicted.

[0208] The codec may consider N intra prediction modes, such as N=35, where a mode may correspond to a direction or a certain angle of an edge, eg, horizontal, vertical, or any other angle.

[0209] In order to provide configurations of the neural network(s) for different intra prediction modes, the neural network architecture may be pre-trained in an offline phase on a certain training dataset, which may not necessarily include the video to be encoded. The neural network may have an architecture that better exploits the availability of information about the intra prediction modes.

[0210] According to an embodiment, the configuration of the neural network may include a mode-agnostic portion of (one or more) layers and a mode-dependent or mode-specific portion of (one or more) layers. In one embodiment, there may be an initial portion of initial layers that are executed or trained for all prediction modes and several final layers specific to each prediction mode.

[0211] According to another embodiment, the configuration of the neural network may include multiple different neural networks for different prediction modes. For example, there may be as many neural networks as prediction models, and each block group may be used to train one of these neural networks. This may add prediction capabilities, but may also increase the memory footprint and computational complexity of the overall system.

[0212] Figure 8The configuration of a neural network according to a first option is shown: a single initial part of initial layers shared by all or multiple prediction modes, and N final layers specific to or dependent on the corresponding prediction mode.

[0213] The network can be trained offline in a manner similar to the fine-tuning process performed in the inference phase. It is assumed that a training data set of images or frames is available. The training images can be input to a traditional codec, such as HEVC, which uses a traditional intra-frame prediction algorithm to generate input blocks, intra-frame prediction blocks, and prediction modes. The input blocks may include previously reconstructed pixels and missing pixels to be predicted. Therefore, the training data may include a set of samples, each of which is a labeled block (i.e., labeled data to be predicted), an input block for intra-frame prediction, an output block predicted by a traditional algorithm, and a prediction mode selected by the traditional algorithm as the best prediction mode among all or some available prediction modes.

[0214] Determine the intra prediction mode for each block (606)

[0215] For the current frame(s), the intra prediction mode for each block may be determined, for example, using (one or more) (algorithmic) methods provided by a video coding standard (e.g., HEVC). This may involve trying a number of different modes (such as predefined prediction modes, smoothly varying prediction modes, or directional modes) and selecting the most suitable mode, for example based on the MSE of the mode between the predicted block and the true block. The selected prediction mode is denoted the labeled mode.

[0216] Label the calculated path (608)

[0217] When a frame is to be encoded as an intra frame, a conventional intra prediction process (e.g., according to the HEVC standard) may be run to determine the computation paths for different intra prediction modes. The computation paths may include information about the relationship between different mode-agnostic layers or mode-dependent layers. For example, to which mode-dependent layer should the output of the mode-agnostic layer be provided. The computation paths may be represented by "computation path annotation labels" or simply "computation path labels". For example, the computation path annotation labels correspond to the intra prediction modes, which may then be used in the neural network optimization process to determine the computation path to be taken by the corresponding block. In other words, in the neural network optimization process, for a particular input block, the intra prediction model is selected by selecting the most suitable computation path (e.g., from multiple mode-agnostic layers to a particular mode-dependent layer) among (one or more) mode-agnostic layers and (one or more) mode-dependent layers. Here, a conventional intra prediction process may be run for each block and by considering all possible prediction modes. Therefore, for each input block, there may be N output prediction blocks. A selection parameter, such as an MSE, is calculated between the input block and each of the N output prediction blocks. An output prediction block having the best value of a selection parameter (such as a lowest MSE value) is selected, and a computation path label (e.g., a tagged computation path (GT-CP label)) corresponding to the intra prediction mode of the selected block may be assigned to a specific input block.

[0218] Fig. 9 The determination of a calculation path (e.g., a GT-CP label) is shown. The input frame shown on the left is divided into input blocks. The process described above of selecting the most suitable intra-frame prediction mode and thereby selecting the most suitable calculation path (e.g., from one or more initial layers to a specific final layer) among (one or more) mode-agnostic layers and mode-dependent layers is performed, wherein a traditional intra-frame prediction process, such as HEVC intra-frame prediction, can be applied. The output frame on the right shows the annotated calculation path (GT-CP) labels generated for all blocks of the intra-frame. In some embodiments, all blocks of (one or more) intra-frames may not be assigned an intra-frame prediction mode or calculation path. For example, the encoder may determine not to encode such blocks or encode them by other means, and therefore, no intra-frame prediction mode or calculation path is assigned. For example, the encoder may indicate which image fragments (e.g., strips or block groups) of a picture are intra-coded, and may additionally indicate which intra-coded image fragments are assigned calculation paths.

[0219] Group blocks with the same intra prediction mode into groups (608)

[0220] The encoder proceeds by grouping blocks assigned to the same prediction mode together under the same prediction mode label. In other words, blocks in the current intra frame that have been assigned the same computation path (e.g., the same annotated computation path label) are included in a common group. These groups will then represent the training data set for optimization, where blocks belonging to the same group are used to train both mode-agnostic layers and mode-dependent layers, such as the initial layer and the final layer associated with a specific GT-CP label.

[0221] Fig.10 shows the grouping process, where Fig. 9 The blocks with GT-CP label value 1 (reference intra prediction mode 1) in the output frame of are grouped together. These blocks or a subset thereof are then used as a training dataset for optimizing the computation path among (one or more) mode-agnostic layers and (one or more) mode-1-dependent layers (e.g., from multiple initial layers to a mode-1-specific final layer).

[0222] According to an embodiment, the grouping of blocks according to the GT-CP label is performed on multiple intra frames rather than on a single intra frame. Therefore, training blocks can be derived from multiple nearby intra frames, thereby providing more training data for each type of block.

[0223] Embodiments have been described above with respect to (one or more) intra frames. It should be understood that embodiments are generally applicable to intra-coded blocks or image segments (such as slices or groups of blocks) of any type of picture. Furthermore, embodiments are generally applicable to any number of frames, i.e., training can be performed from multiple frames. For example, training can be performed on the basis of intra-coded blocks of refresh regions of a progressive decoding refresh cycle of a frame.

[0224] Training multiple initial layers and specific final layers of a neural network (610)

[0225] According to an embodiment, training includes using the reconstructed pixels on the top row and left column relative to the area to be predicted as input, and using the original content of the area to be predicted as annotations for predicting training loss. For example, the training loss can be the MSE between the output of the neural network (i.e., the prediction) and the annotated block content. Note that in this article, annotating block content and annotating computation path labels represent different things: annotating block content is used to calculate training loss, while annotating computation path labels are used to decide the computation path; in other words, which final layer to use during training.

[0226] According to an embodiment, a training cycle of an initial layer and a specific final layer is performed, in particular as an intra prediction mode. For example, first blocks of mode 1 are used to train mode-agnostic layers and mode-1 specific layers, then blocks of mode 2 are used to train mode-agnostic layers and mode-2 specific layers, and so on, until all modes or a predetermined subset of modes have been considered. This can occur within one training cycle, i.e., until all training data has been used, and then a new training cycle can be performed.

[0227] According to another embodiment, the training of the mode-agnostic layer and the specific mode-dependent final layer involves multiple intra-frame prediction modes at the same time. Therefore, the training data can be shuffled so that multiple modes can be considered for training the neural network at each training iteration. In other words, the training batch can contain data from different modes.

[0228] According to an embodiment, training includes using a lower learning rate for mode-agnostic layers and mode-dependent layers relative to the learning rate used in the offline pre-training phase. Examples of the method are described herein for multiple initial mode-agnostic layers and mode-specific final layers. The learning rate for the mode-specific final layer can be different from the learning rate for the initial layer. For example, during pre-training, an initial learning rate of 0.1 can be used, wherein the initial learning rate can be ultimately reduced during pre-training, for example using any known learning rate decay technique. Thereafter, during training of the initial layers, an initial learning rate of 0.001 can be used, and during training of the mode-specific layers, an initial learning rate of 0.0001 can be used.

[0229] According to an embodiment, training includes using all reconstructed pixels or a subset of reconstructed pixels. When using all reconstructed pixels, the neural network may also include a recursive layer that allows for input data of variable size. Using all reconstructed pixels may be beneficial because it provides the neural network with more context on which to base predictions.

[0230] According to an embodiment, training includes using areas to be predicted that are filled with values ​​(such as zeros). This can be beneficial for neural networks that include convolutional layers because the spatial relationship between the reconstructed pixels and the pixels to be predicted will be preserved.

[0231] According to an embodiment, training includes using a region to be predicted that has been filled with (one or more) values ​​output by another algorithm, such as a prediction from another non-learning algorithm (e.g., traditional intra-frame prediction performed by HEVC or other codecs) or a learning algorithm (e.g., a neural network). This can be beneficial in particular for models that do not have a large receptive field in their last layers. In the case where the region to be predicted is filled with (one or more) values ​​from the intra-frame prediction of a traditional codec, this is easily available from the labeling of the calculation path. In other words, the intra-frame prediction of the traditional codec can be performed only once for a certain block, and then it can be used to obtain the GT-CP label and for using the prediction as an additional input to the intra-frame prediction neural network. In particular, the additional input of the neural network can be a prediction output by the traditional codec using a pattern corresponding to the previously determined GT-CP label.

[0232] According to an embodiment, if grouping and adaptation of blocks according to GT-CP labels are performed on a single intra frame, the previously adapted neural network is used to perform intra prediction on subsequent intra frames.

[0233] According to an embodiment, if grouping and adaptation of blocks according to the GT-CP tag are performed on a plurality of intra frames instead of a single intra frame, intra prediction for the plurality of frames is performed using an adapted network.

[0234] According to an embodiment, training is performed on intra frames at intervals of a predetermined number of consecutive intra frames. Therefore, the neural network adaptation process only needs to be performed once every few intra frames, because it is assumed that the image content of each block will not change much in a short time.

[0235] According to an embodiment, training is performed for each intra frame, wherein the training data is sent to the decoder if a predetermined accuracy improvement is achieved. Thus, if the adapted neural network provides sufficient accuracy gain relative to the bitrate overhead, the encoder can continuously perform adaptation and decide to send (higher quality) training blocks. Thus, if the gain is sufficient in terms of rate-distortion trade-off, the encoder accepts the network adaptation process. When the network adaptation process is accepted, the encoder encodes high-quality blocks for training into the bitstream so that the decoder can also adapt his network and align with the encoder. If the network adaptation process is rejected, the encoder will return to the version of the neural network before the adaptation process.

[0236] According to an embodiment, training includes using a subset of the blocks of the current intra frame for adaptation. For example, the encoder can be configured to use only a subset of the blocks of the current intra frame to keep the class (ie, mode) balanced, i.e. have a similar number of blocks for each mode.

[0237] According to an embodiment, training includes using a subset of blocks for training (one or more) intra-prediction mode agnostic layers and (one or more) intra-prediction mode dependent layers (e.g., all initial layers and all final layers) without using any labeled computation path labels. The blocks in the subset pass through the initial layer, then all final layers, and the loss is calculated from the final layer, differentiated relative to the parameters of the network, and finally the network parameters are updated according to the calculated gradients. This embodiment is beneficial in cases where the labeling of the computation path is too expensive to be performed on all blocks (in the current frame or in multiple frames, depending on which option is selected). Therefore, a subset of blocks is marked and used to adapt the initial layer and the mode-specific final layer, while another subset of blocks is not marked and used to adapt the initial layer and the final layer. The learning rates for the two subsets may be different, and the learning rates for training the (one or more) initial layers and the (one or more) final layers may also be different. For example, for a subset of unlabeled blocks, the initial layer(s) may be trained with a smaller learning rate (e.g., 10 times smaller) than the learning rate used when the blocks were labeled, and the final layer(s) may be trained with an even smaller learning rate (e.g., 10 times smaller than the learning rate used for the initial layer for the same training process).

[0238] According to an embodiment, training includes using a subset of the patterns in the current frame for adaptation. Thereafter, another subset of the patterns may be used in subsequent frames. This may be beneficial, for example, when certain patterns are missing in a certain frame but present in future frames.

[0239] Information indicating a block group defined as a training block (704)

[0240] The encoder indicates information indicating which blocks are training blocks in or along the bitstream. The decoder decodes the information indicating which blocks are training blocks from or along the bitstream and performs fine tuning of the neural network accordingly. According to an embodiment, the information indicating which blocks are training blocks may include, but is not limited to, one or more of the following:

[0241] - A flag for each block, which informs the decoder that the block is a training block. The flag may be arithmetically coded, and the arithmetic coding may be context adaptive (e.g., based on the value of the flag for neighboring blocks and / or the quantization parameter value for the block). Thus, the flag may be represented by a fraction of bits in the bitstream. For example, the flag (which may be arithmetically coded) may be included in a coding tree unit syntax structure, which may be referred to, for example, as coding_tree_unit() in the syntax definition.

[0242] - A flag for an image segment (such as a slice or block group) that may be included in the header syntax structure for the image segment. When set, the flag may indicate that intra-coded blocks of the image segment are training blocks. In some cases, the flag may be conditionally present only if the image segment is indicated as being intra-coded or intra-predicted.

[0243] - A threshold quantization parameter (QP) value indicating the maximum QP for a block to qualify as a training block.

[0244] - Regions within a picture are indicated as including training blocks, while other regions of the picture may not contain training blocks. Regions may be indicated, for example, as width and height in blocks starting from the top left corner of the picture, or in any other suitable way.

[0245] If the block is a training block, the decoder can use it for adapting the intra prediction neural network. In some cases, the decoder can determine whether a block is a "training block" by checking the quality level (quantization parameter) and by comparing the quality level to an indicated or inferred threshold. The threshold can be inferred, for example, by calculating an average quality level, which can be common for all prediction modes or specific for each prediction mode. The threshold will then be the calculated average, or the calculated average plus a predefined value (or a percentage of the calculated average).

[0246] According to an embodiment, the method further includes using a decoded version of the block to be predicted as annotation data in the encoder and the decoder. In order to keep the intra-frame prediction neural network aligned in the encoder and the decoder and have the same parameter values, they need to be trained using the same input and the same annotation data. The input can be considered as the reconstructed data above and to the left of the block to be predicted, and the annotation data is the true content of the block to be predicted. However, the decoder is not provided with the exact true content of the block to be predicted, but the data of the block is available on the decoder side with only some encoding artifacts (e.g., quantization errors). Therefore, the encoder needs to use the same data, i.e., the decoded block. The block can be obtained, for example, by encoding and decoding the block in intra-frame mode using traditional prediction (e.g., HEVC) and high encoding quality (e.g., setting a low quantization parameter for encoding the residual). Alternatively, the current version of the neural network can be used for intra-frame prediction, where high encoding quality is applied (e.g., setting a low quantization parameter for encoding the residual).

[0247] In the above, the embodiments are described as block-by-block operations, but it should be understood that all operations or subsets of operations may be performed for other units (such as (one or more) complete pictures or individual pixels). Therefore, it should be further noted that blocks (such as input blocks, prediction blocks, prediction error blocks, and reconstructed prediction error blocks) may also be referred to as signals (such as input signals, prediction signals, prediction error signals, and reconstructed prediction error signals) in the specification and claims.

[0248] Fig.11 It is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. Data source 1510 provides a source signal in an analog, uncompressed digital, or compressed digital format or any combination of these formats. Encoder 1520 may include or be connected with pre-processing, such as data format conversion and / or filtering of the source signal. Encoder 1520 encodes the source signal into a coded media bitstream. It should be noted that the bitstream to be decoded can be directly or indirectly received from a remote device located in a network of virtually any type. In addition, the bitstream can be received from local hardware or software. Encoder 1520 may be able to encode more than one media type, such as audio and video, or more than one encoder 1520 may be required to encode the source signal of different media types. Encoder 1520 may also obtain synthetically generated input, such as graphics and text, or it may be able to produce a coded bitstream of synthetic media. The processing of a coded media bitstream of only one media type is considered below to simplify the description. However, it should be noted that usually real-time broadcast services include several streams (usually at least one audio, video and text subtitle stream). It should also be noted that the system may include many encoders, but only one encoder 1520 is presented in the figure to simplify the description without loss of generality. It should be further understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles also apply to the corresponding decoding process, and vice versa.

[0249] The coded media bitstream may be transmitted to a storage device 1530. The storage device 1530 may include any type of mass storage to store the coded media bitstream. The format of the coded media bitstream in the storage device 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file, or the coded media bitstream may be encapsulated into a Segment format suitable for DASH (or a similar streaming system) and stored as a Segment sequence. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store one or more media bitstreams in a file and create file format metadata, which may also be stored in the file. The encoder 1520 or the storage device 1530 may include a file generator, or the file generator may be operably attached to the encoder 1520 or the storage device 1530. Some systems operate "live", i.e., the storage device is omitted, and the coded media bitstream is transmitted directly from the encoder 1520 to the transmitter 1540. Then, as needed, the coded media bitstream may be transmitted to the transmitter 1540 (also referred to as a server). The format used in the transmission may be a basic self-contained bitstream format, a packet stream format, a Segment format suitable for DASH (or similar streaming systems), or one or more coded media bitstreams may be encapsulated into a container file. The encoder 1520, the storage device 1530, and the server 1540 may reside in the same physical device, or they may be included in different devices. The encoder 1520 and the server 1540 may operate real-time content, in which case the coded media bitstream is typically not stored permanently, but is buffered for a short period of time in the content encoder 1520 and / or the server 1540 to eliminate processing delays, transmission delays, and changes in the coded media bitrate.

[0250] Server 1540 uses a communication protocol stack to send a coded media bitstream. The stack may include, but is not limited to, one or more of the following: Real-time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). When the communication protocol stack is packet-oriented, server 1540 encapsulates the coded media bitstream into packets. For example, when using RTP, server 1540 encapsulates the coded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system may include more than one server 1540, but for simplicity, the following description only considers one server 1540.

[0251] If the media content is encapsulated in a container file for use in a storage device 1530 or for inputting data to a transmitter 1540, the transmitter 1540 may include or be operably attached to a "send file parser" (not shown in the figure). In particular, if the container file is not sent as such but at least one of the contained coded media bitstreams is encapsulated for transmission via a communication protocol, the send file parser locates the appropriate portion of the coded media bitstream to be transmitted via the communication protocol. The send file parser may also help create the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions (such as a hint track in ISOBMFF) for encapsulating at least one of the contained media bitstreams on the communication protocol.

[0252] Server 1540 may or may not be connected to gateway 1550 via a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. A gateway may also or alternatively be referred to as a middlebox. For DASH, a gateway may be an edge server (of a CDN) or a network proxy. Note that a system may typically include any number of gateways, etc., but for simplicity's sake, the following description only considers one gateway 1550. Gateway 1550 may perform different types of functions, such as conversion of a packet stream from one communication protocol stack to another, merging and bifurcation of data streams, and manipulation of data streams according to downlink and / or receiver capabilities (such as controlling the bit rate of a forwarded stream according to the main downlink network conditions). In various embodiments, gateway 1550 may be a server entity.

[0253] The system includes one or more receivers 1560, which are generally capable of receiving, demodulating and decapsulating the transmitted signal into a coded media bitstream. The coded media bitstream can be transmitted to a recording storage device 1570. The recording storage device 1570 may include any type of mass storage to store the coded media bitstream. The recording storage device 1570 may alternatively or additionally include a computing memory, such as a random access memory. The format of the coded media bitstream in the recording storage device 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. If there are multiple coded media bitstreams associated with each other (such as audio streams and video streams), a container file is generally used, and the receiver 1560 includes or is attached to a container file generator, which generates a container file from an input stream. Some systems operate "live", i.e., the recording storage device 1570 is omitted, and the coded media bitstream is transmitted directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recorded stream, such as an excerpt of the most recent 10 minutes of the recorded stream, is maintained in the recording storage device 1570 , while any earlier recorded data is discarded from the recording storage device 1570 .

[0254] The coded media bitstream may be transferred from the recording storage device 1570 to the decoder 1580. If there are multiple coded media bitstreams (such as audio and video streams) that are associated with each other and encapsulated into a container file, or a single media bitstream is encapsulated in a container file, for example, for easier access, a file parser (not shown in the figure) is used to decapsulate each coded media bitstream from the container file. The recording storage device 1570 or the decoder 1580 may include the file parser, or the file parser is attached to the recording storage device 1570 or the decoder 1580. It should also be noted that the system may include many decoders, but only one decoder 1570 is discussed here to simplify the description without loss of generality.

[0255] The coded media bitstream may be further processed by a decoder 1570, the output of which is one or more decompressed media streams. Finally, a renderer 1590 may reproduce the decompressed media streams using, for example, a speaker or display. The receiver 1560, the recording storage device 1570, the decoder 1580, and the renderer 1590 may reside in the same physical device, or they may be included in different devices.

[0256] The transmitter 1540 and / or the gateway 1550 may be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bitrate adaptation and / or fast start, and / or the transmitter 1540 and / or the gateway 1550 may be configured to select the representation (one or more) sent. The switching between different representations may occur for a variety of reasons, such as in response to a request from the receiver 1560 or the prevailing conditions (such as throughput) of the network on which the bitstream is transmitted. In other words, the receiver 1560 may initiate switching between representations. The request from the receiver may be, for example, a request for a Segment or Subsegment from a different representation than before, a request for a change in the scalability layer and / or sublayer sent, or a change in a rendering device with different capabilities compared to the previous capabilities. The request for a Segment may be an HTTP GET request. The request for a Subsegment may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, to provide so-called fast start in streaming services, where the bitrate of the transmitted stream is below the channel bitrate after start or random access streaming, so as to start playback immediately and reach a buffer occupancy level that allows for occasional packet delays and / or retransmissions. The bitrate adaptation may include multiple representation or layer up-switching operations and representation or layer down-switching operations occurring in various orders.

[0257] Decoder 1580 can be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast start, and / or decoder 1580 can be configured to select the (one or more) representations sent. The switching between different representations can occur for a variety of reasons, such as to achieve faster decoding operation or to adapt the transmitted bitstream, for example, in terms of bitrate, to the main conditions (such as throughput) of the network on which the bitstream is transmitted. For example, if the device including decoder 1580 is multitasking and uses computing resources for other purposes except decoding video bitstream, faster decoding operation may be required. In another example, when content is played back at a speed faster than normal playback speed (for example, twice or three times faster than traditional real-time playback rate), faster decoding operation may be required.

[0258] In the above, some embodiments have been described with reference to MSE (e.g., as an example of a selection criterion for a prediction mode). It should be understood that other metrics, such as the sum of absolute differences (SAD), the sum of squared differences (SSD), the sum of absolute transformed differences (SATD), can be used similarly. Similarly, the Lagrangian cost function combination rate and any distortion metric can be used similarly.

[0259] In the above, some embodiments have been described with reference to and / or using the terminology of HEVC. It should be understood that the embodiments can be similarly implemented with any video encoder and / or video decoder.

[0260] The embodiments of the invention described above describe the codec in terms of separate encoder devices and decoder devices to help understand the processes involved. However, it will be appreciated that the devices, structures, and operations can be implemented as a single encoder-decoder device / structure / operation. In addition, the encoder and decoder can share some or all common elements.

[0261] Although the above examples describe embodiments of the present invention operating within a codec within an electronic device, it will be appreciated that the present invention as defined in the claims may be implemented as part of any video codec. Thus, for example, embodiments of the present invention may be implemented in a video codec that may implement video encoding via a fixed or wired communication path.

[0262] Thus, the user equipment may comprise a video codec, such as the video codecs described in the embodiments of the invention above.It should be understood that the term "user equipment" is intended to cover any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device or a portable web browser.

[0263] Furthermore, elements of a public land mobile network (PLMN) may also include a video codec as described above.

[0264] Generally, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0265] Embodiments of the present invention can be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. Further, in this regard, it should be noted that any box of the logic flow in the figure can represent a program step, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software can be stored in a physical medium such as a memory chip or a storage block implemented in a processor, a magnetic medium such as a hard disk or a floppy disk, and an optical medium such as, for example, a DVD and its data variant CD.

[0266] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as non-limiting examples.

[0267] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0268] Programs such as those provided by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and position components on a semiconductor chip. Once a design for a semiconductor circuit has been completed, the final design in a standardized electronic format (e.g., Opus, GDSII, etc.) is sent to a semiconductor manufacturing facility or "fab" for fabrication.

[0269] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. Nevertheless, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention.

Claims

1. A method for video encoding, comprising: obtaining a configuration of at least one neural network, the configuration comprising a plurality of intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers, the plurality of intra-prediction mode specific layers corresponding to different intra-prediction modes; obtaining at least one input video frame comprising a plurality of blocks; Determining to encode one or more blocks using intra prediction; determining an intra prediction mode for each of the one or more blocks; grouping blocks having the same intra prediction mode into groups, the groups being assigned computation paths from outputs of at least one intra prediction mode agnostic layer of the plurality of intra prediction mode agnostic layers to inputs of the one or more intra prediction mode specific layers; training the plurality of intra-prediction mode agnostic layers and / or the one or more intra-prediction mode specific layers of the at least one neural network based on a training loss between an output of the at least one neural network related to a group of blocks having a same intra-prediction mode and an annotated block, wherein the annotated block comprises at least one of a block of the input video frame or a reconstructed block; as well as The block is encoded using the computation path assigned to the intra prediction mode for the block.

2. The method according to claim 1, wherein: The configuration of the at least one neural network includes a pre-trained neural network trained using a group of blocks having the same intra prediction mode, and the method further includes: A block of the input video frame is used as a labeled block.

3. The method according to claim 1, further comprising: Information indicating a group of blocks defined as training blocks is indicated in or along a bitstream comprising the group of coded blocks.

4. A method for video decoding, comprising: obtaining a configuration of at least one neural network, the configuration comprising a plurality of intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers, the plurality of intra-prediction mode specific layers corresponding to different intra-prediction modes; Obtaining a bitstream comprising an encoded video frame, wherein the encoded video frame comprises a plurality of blocks; decoding from or along the bitstream information indicating a group of blocks having the same intra prediction mode defined as training blocks; Decoding the training block into a reconstructed block; training the plurality of intra-prediction mode agnostic layers and / or the one or more intra-prediction mode specific layers of the at least one neural network based on a training loss between an output of the at least one neural network related to a group of blocks having the same intra-prediction mode and the reconstructed block; and The coding block is decoded using a computation path of the intra prediction mode assigned to the coding block from an output of at least one of the plurality of intra prediction mode agnostic layers to an input of the one or more intra prediction mode specific layers.

5. The method according to claim 4, wherein: The configuration of the at least one neural network comprises a pre-trained neural network trained using a group of blocks having the same intra prediction mode, and the method further comprises: The reconstructed block is used as the annotation block.

6. An apparatus for video encoding, comprising: means for obtaining a configuration of at least one neural network, the configuration comprising a plurality of intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers, the plurality of intra-prediction mode specific layers corresponding to different intra-prediction modes; means for obtaining at least one input video frame comprising a plurality of blocks; means for determining to encode one or more blocks using intra prediction; means for determining an intra prediction mode for each of the one or more blocks; means for grouping blocks having the same intra prediction mode into groups, the groups being assigned with computation paths from outputs of at least one intra prediction mode agnostic layer of the plurality of intra prediction mode agnostic layers to inputs of the one or more intra prediction mode specific layers; training components of the plurality of intra-prediction mode agnostic layers and / or the one or more intra-prediction mode specific layers of the at least one neural network based on a training loss between an output of the at least one neural network related to a group of blocks having the same intra-prediction mode and an annotated block, wherein the annotated block comprises at least one of a block of the input video frame or a reconstructed block; as well as Means for encoding the block using the computation path assigned to the intra prediction mode for the block.

7. The device according to claim 6, wherein: The configuration of the at least one neural network comprises a pre-trained neural network trained using a group of blocks having the same intra prediction mode, and the apparatus further comprises: Means for using a block of said input video frame as a labeled block.

8. The apparatus according to claim 6, further comprising: Means for indicating in or along a bitstream comprising a group of coding blocks information indicating a group of blocks defined as training blocks.

9. The device according to any one of claims 6 to 8, wherein: The intra prediction mode agnostic layer includes initial layers trained for a plurality of intra prediction modes, and the intra prediction mode specific layer includes a plurality of final layers specific to an intra prediction mode.

10. The device according to any one of claims 6 to 8, wherein: The configuration of the at least one neural network includes multiple neural networks for different intra-frame prediction modes.

11. The device according to any one of claims 6 to 8, wherein: A first learning rate is used to train the intra prediction mode agnostic layer, a second learning rate is used to train the intra prediction mode specific layer, and wherein the second learning rate is less than the first learning rate.

12. The device according to claim 8, wherein: The information indicating the group of blocks defined as training blocks includes a quantization parameter, wherein if the quantization parameter is less than or equal to a threshold value, the group of blocks is defined as a training block.

13. The device according to any one of claims 6 to 8, wherein: The input to the at least one neural network comprises a plurality of reconstructed blocks of the input video frame.

14. The device according to any one of claims 6 to 8, wherein: The epochs or iterations of training the plurality of intra prediction mode agnostic layers and / or the one or more intra prediction mode specific layers are based on blocks associated with a plurality of intra prediction modes.

15. An apparatus for video decoding, comprising: means for obtaining a configuration of at least one neural network, the configuration comprising a plurality of intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers, the plurality of intra-prediction mode specific layers corresponding to different intra-prediction modes; Means for obtaining a bitstream comprising an encoded video frame, wherein the encoded video frame comprises a plurality of blocks; means for decoding from or along the bitstream information indicating a group of blocks having the same intra prediction mode defined as training blocks; means for decoding said training block into a reconstructed block; means for training the plurality of intra-prediction mode agnostic layers and / or the one or more intra-prediction mode specific layers of the at least one neural network based on a training loss between an output of the at least one neural network related to a group of blocks having the same intra-prediction mode and the reconstructed block; and Means for decoding a coding block using a computation path of an intra prediction mode assigned to the coding block from an output of at least one of the plurality of intra prediction mode agnostic layers to an input of the one or more intra prediction mode specific layers.

16. The device according to claim 15, wherein: The configuration of the at least one neural network comprises a pre-trained neural network trained using a group of blocks having the same intra prediction mode, and the apparatus further comprises: Means for using the reconstructed block as a labeled block.

17. The device according to any one of claims 15 to 16, wherein: The intra prediction mode agnostic layer includes initial layers trained for a plurality of intra prediction modes, and the intra prediction mode specific layer includes a plurality of final layers specific to an intra prediction mode.

18. The device according to any one of claims 15 to 16, wherein: The configuration of the at least one neural network includes multiple neural networks for different intra-frame prediction modes.

19. The device according to any one of claims 15 to 16, wherein: A first learning rate is used to train the intra prediction mode agnostic layer, a second learning rate is used to train the intra prediction mode specific layer, and wherein the second learning rate is less than the first learning rate.

20. The device according to any one of claims 15 to 16, wherein: The information indicating the group of blocks defined as training blocks includes a quantization parameter, wherein if the quantization parameter is less than or equal to a threshold value, the group of blocks is defined as a training block.

21. The device according to any one of claims 15 to 16, wherein: The epochs or iterations of training the plurality of intra prediction mode agnostic layers and / or the one or more intra prediction mode specific layers are based on blocks associated with a plurality of intra prediction modes.

Images