Method and apparatus for encoding and decoding video data, computer readable storage medium and computer program product

Abstract

The present disclosure relates to a method and an apparatus for encoding and decoding video data, a computer readable storage medium and a computer program product. A bitstream includes video data corresponding to one or more slices. The bitstream includes a picture header including syntax elements to be used when decoding the one or more slices, and a slice header including syntax elements to be used when decoding the slice. Decoding includes parsing at least one syntax element in the picture header indicating whether a decoding tool can be used in the picture. When the decoding tool is used in the picture, at least one APS ID related syntax element is parsed in the picture header for the decoding tool. Decoding further includes parsing at least one syntax element in the slice header before syntax elements related to other decoding tools, indicating whether the decoding tool is to be used for the slice. The bitstream is decoded using the syntax elements.

Description

[0001] (This application is a divisional application of the application filed on February 26, 2021, with application number 202180018944X, entitled "Method and apparatus for encoding and decoding video data, computer-readable storage medium and computer program product".) Technical Field

[0002] This invention relates to video encoding and decoding, and more particularly to high-level syntax for bitstreams. Background Technology

[0003] Recently, the Joint Video Experts Group (JVET) (a collaborative team comprised of MPEG and ITU-T Study Group 16 VCEG) began working on a new video coding standard called Multifunctional Video Coding (VVC). VVC aims to provide a significant improvement in compression performance (i.e., typically twice as much) over the existing HEVC standard and is expected to be finalized by 2020. Key target applications and services include, but are not limited to, 360-degree and High Dynamic Range (HDR) video. In summary, JVET evaluated feedback from 32 organizations using formal subjective testing conducted by independent test laboratories. Some recommendations indicate that compression efficiency is typically improved by 40% or more compared to HEVC. Specific effects were shown on Ultra High Definition (UHD) video test materials. Therefore, for the final standard, we can expect an improvement in compression efficiency far exceeding the target of 50%.

[0004] The JVET Exploratory Model (JEM) uses all HEVC tools and has introduced several new ones. These changes require altering the structure of the bitstream, particularly the high-level syntax that can affect the overall bit rate of the bitstream.

[0005] A significant change in advanced syntax is the introduction of "picture headers" into the bitstream. A picture header is a header element used to specify the syntax elements to be used when decoding individual stripes within a particular picture (or frame). Therefore, the picture header is placed before the stripe-related data in the bitstream, and each stripe has its own "strip header." See below for reference. Figure 6 Describe the structure in more detail.

[0006] At the 16th meeting (October 1-11, 2019, Geneva, Switzerland), document JVET-P0239 entitled “AHG 17: PictureHeader” proposed the introduction of mandatory picture headers into VVC, and this was adopted as Universal Video Coding (Draft 7) and uploaded as document JVET_P2001.

[0007] However, this header has a large number of parameters, all of which need to be parsed in order to use any particular encoding tool. Summary of the Invention

[0008] This invention relates to an improvement in the structure of the image header to simplify the parsing process, which reduces complexity without any reduction in encoding performance.

[0009] In particular, by setting syntactic elements related to APS ID information at the beginning of the image header, these elements can be parsed first, which eliminates the need to parse the rest of the header.

[0010] Similarly, if there are syntax elements related to APS ID information in the strip header, these syntax elements are set at the beginning of the strip header.

[0011] In one example, it is proposed to move the syntactic elements associated with the APS ID in the early stages of the image header and stripe header. The aim of this modification is to reduce the parsing complexity for some streaming applications that need to track the APS ID in the image header and stripe header to remove unused APSs. The proposed modification has no impact on BDR performance.

[0012] This reduces the parsing complexity for streaming applications, where the APS ID information can be all the information needed in the header. For the same reason, other streaming-related syntax elements can be moved towards the top of the header.

[0013] It should be understood that the term "start" does not necessarily mean the first entry in the corresponding header, as there may be multiple introductory syntax elements preceding the syntax elements related to the APS ID information. The detailed descriptions illustrate various examples, but the general definition is that the syntax elements related to the APS ID information are provided before the syntax elements related to the decoding tool. In a specific example, the syntax elements related to the APS ID of the ALF, LMCS, and scaling list are placed exactly after the poc_msb_val syntax element.

[0014] According to a first aspect of the invention, a method is provided for decoding video data from a bitstream, the bitstream comprising video data corresponding to one or more stripes. The bitstream includes an image header and a stripe header, the image header including syntax elements to be used when decoding one or more stripes, and the stripe header including syntax elements to be used when decoding a stripe. The decoding includes parsing at least one syntax element in the image header indicating whether a decoding tool or parameter can be used in the image, wherein, when the decoding tool is used in the image, at least one APS ID-related syntax element is parsed in the image header for the decoding tool or parameter. The decoding also includes parsing at least one syntax element in the stripe header, before other syntax elements related to decoding tools or parameters, indicating whether the decoding tool or parameter should be used for that stripe, and using the syntax element to decode the bitstream.

[0015] Therefore, information related to enabling or disabling decoding tools or parameters at the stripe level (such as Luminance Map with Chroma Scaling (LMCS) or scaling lists) is set at or near the beginning of the stripe header. This enables a simpler and faster parsing process, especially for streaming applications.

[0016] Parameters for another (alternative) decoding tool, or parameters containing APS ID-related syntax elements, can be parsed when enabled, prior to syntax elements indicating whether a decoding tool or parameter should be used for that stripe (e.g., prior to decoding tools or parameters related to LMCS or scaling lists). APS ID-related syntax elements can be associated with adaptive loop filtering APSIDs.

[0017] This reduces complexity for streaming applications that need to track APS ID usage to remove unused APS NAL units. While APS IDs associated with certain decoding tools and parameters (e.g., LMCS or scaling lists) may not exist within the slice header (e.g., syntactic elements for APS IDs associated with LMCS or scaling lists are not parsed in the slice header), disabling such tools or parameters at the slice level will affect the APS IDs used for the current image. For example, in a sub-image extraction application, the APS ID should be sent in the image header, but the extracted sub-image will only contain one slice. Within this single slice, decoding tools or parameters (e.g., LMCS or scaling lists) can be disabled in the slice header. If an APS identified in the image header is never used in another frame, the extraction application should remove the APS with the associated APS ID (e.g., LMCS or scaling list APS) because the extracted sub-image does not require it. Therefore, a decision on whether to remove an APS can be made efficiently without first parsing other data in the slice header.

[0018] Syntactic elements in the strip header that indicate whether decoding tools or parameters should be used can follow immediately after syntax elements related to Adaptive Loop Filtering (ALF) parameters.

[0019] Decoding tools or parameters can be associated with Luminance Mapping with Chroma Scaling (LMCS). Syntactic elements indicating whether to use decoding tools or parameters can be flags that signal whether to use LMCS for a stripe.

[0020] Decoding tools or parameters can be associated with scaling lists. Syntactic elements indicating whether to use decoding tools or parameters can be flags that signal whether to use scaling lists for a stripe.

[0021] In an embodiment, a method is provided for decoding video data from a bitstream, the bitstream including video data corresponding to one or more stripes, wherein the bitstream includes an image header and a stripe header, the image header including syntax elements to be used when decoding one or more stripes, and the stripe header including syntax elements to be used when decoding stripes, the method comprising: parsing in the image header at least one syntax element indicating whether an LMCS or scaling list decoding tool can be used in the image, wherein when an LMCS or scaling list decoding tool is used in the image, at least one APS ID-related syntax element is parsed in the image header for the LMCS or scaling list decoding tool; parsing in the stripe header at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe before other decoding tool-related syntax elements, wherein the ALF APS ID syntax element is parsed when enabled before at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe; and decoding video data from the bitstream using the syntax elements.

[0022] According to a second aspect of the invention, a method is provided for encoding video data into encoded video data in a bitstream, the bitstream comprising video data corresponding to one or more stripes. The bitstream includes a header and a stripe header, the header including syntax elements to be used when decoding one or more stripes, and the stripe header including syntax elements to be used when decoding a stripe. The encoding includes: encoding in the image header at least one syntax element indicating whether a decoding tool or parameter can be used in the image, wherein, when the decoding tool or parameter can be used in the image, at least one APS ID-related syntax element is encoded in the image header for the decoding tool or parameter; and in the stripe header, encoding in the image header at least one syntax element indicating whether the decoding tool or parameter should be used for the stripe, prior to syntax elements related to other decoding tools or parameters.

[0023] The parameters of another (additional) decoding tool that contains APS ID-related syntax elements can be encoded when enabled, before the syntax elements indicating whether to use a decoding tool or parameters for that stripe (e.g., before the decoding tool or parameters associated with LMCS or scaling lists).

[0024] Another decoding tool's APS ID-related syntax elements can be related to the adaptive loop filter APS ID.

[0025] This processing encodes the bitstream, reducing complexity for streaming applications that need to track APS ID usage to remove unused APS NAL units. While APS IDs associated with certain decoding tools and parameters (e.g., LMCS or scaling lists) are not present in the slice header (e.g., the syntax elements of APS IDs associated with LMCS or scaling lists are not encoded in the slice header), disabling such decoding tools or parameters at the slice level will affect the APS ID used for the current image. For example, in a sub-image extraction application, the APS ID should be sent in the image header, but the extracted sub-image will only contain one slice. In this single slice, decoding tools or parameters (e.g., LMCS or scaling lists) can be disabled in the slice header. If an APS identified in the image header is never used in another frame, the extraction application should remove the APS with the associated APS ID (e.g., LMCS or scaling list APS) because the extracted sub-image does not require it. Therefore, a decision on whether to remove an APS can be made efficiently without first parsing other data in the slice header.

[0026] Syntactic elements in the strip header that indicate whether to use decoding tools or parameters can be encoded as immediately following ALF parameters.

[0027] Decoding tools or parameters can be associated with LMCS. Encoded syntax elements indicating whether to use decoding tools or parameters can be flags that signal whether to use LMCS for a stripe.

[0028] Decoding tools or parameters can be associated with scaling lists. Encoded syntax elements indicating whether to use decoding tools or parameters can be flags that signal whether to use scaling lists for a stripe.

[0029] In an embodiment according to the second aspect, a method is provided for encoding video data into a bitstream, the bitstream comprising video data corresponding to one or more stripes, wherein the bitstream comprises an image header and a stripe header, the image header comprising syntax elements to be used when decoding one or more stripes, and the stripe header comprising syntax elements to be used when decoding stripes, the method comprising: encoding in the image header at least one syntax element indicating whether an LMCS or scaling list decoding tool can be used in the image, wherein when the LMCS or scaling list decoding tool is used in the image, at least one APS ID-related syntax element is encoded in the image header for the LMCS or scaling list decoding tool; in the stripe header, before syntax elements related to other decoding tools, encoding at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe, wherein the ALF APS ID syntax element, when enabled, is encoded before the at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe; and encoding video data into a bitstream using the syntax elements.

[0030] According to a third aspect of the invention, a decoder is provided for decoding video data from a bitstream, the decoder being configured to perform any implementation of the method according to the first aspect.

[0031] According to a fourth aspect of the invention, an encoder is provided for encoding video data into a bitstream, the encoder being configured to perform any implementation of the method according to the second aspect.

[0032] According to a fifth aspect of the invention, a computer program is provided that, when executed, causes the methods of either the first or second aspect to be performed. The program may be provided separately, or may be carried on, by, or in a carrier medium. The carrier medium may be non-transitory, such as a storage medium, particularly a computer-readable storage medium. The carrier medium may also be transient, such as a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet.

[0033] According to a sixth aspect of the invention, a method for parsing a bitstream is provided, the bitstream comprising video data corresponding to one or more stripes. The bitstream includes an image header and a stripe header, the image header including syntax elements to be used when decoding one or more stripes, and the stripe header including syntax elements to be used when decoding a stripe, comprising: parsing at least one syntax element in the image header indicating whether a decoding tool or parameter can be used in the stripe, wherein, when the decoding tool or parameter is used in the image, at least one APS ID-related syntax element is parsed in the image header for the decoding tool; and in the stripe header, at least one syntax element indicating whether the decoding tool or parameter should be used for the stripe is not parsed before other syntax elements related to the decoding tool or parameter.

[0034] This method reduces complexity compared to setting flags or information at the stripe level for decoding tools or parameters (LMCS or scaling list) because it does not require parsing the stripe header for LMCS.

[0035] Decoding tools or parameters may be associated with a luminance map with chroma scaling (LMCS). Alternatively or additionally, decoding tools or parameters may be associated with a scaling list.

[0036] According to a seventh aspect of the invention, a method for streaming image data is provided, comprising parsing a bitstream according to a sixth aspect.

[0037] In the eighth aspect, an apparatus configured to perform the method according to the sixth aspect is provided.

[0038] In a ninth aspect, a computer program is provided that, when executed, causes the method of the sixth aspect to be performed. The program may be provided separately, or may be carried on, by, or in a carrier medium. The carrier medium may be non-transitory, such as a storage medium, particularly a computer-readable storage medium. The carrier medium may also be transient, such as a signal or other transmission medium. The signal may be transmitted via any suitable network, including the Internet. Other features of the invention are characterized by the independent and dependent claims.

[0039] In a tenth aspect, a bitstream is provided, the bitstream comprising video data corresponding to one or more stripes, wherein the bitstream comprises an image header and a stripe header, the image header comprising syntax elements to be used when decoding one or more stripes, the stripe header comprising syntax elements to be used when decoding stripes, the bitstream encoding in the image header at least one syntax element indicating whether an LMCS or scaling list decoding tool can be used in the image, wherein when an LMCS or scaling list decoding tool is used in the image, at least one APS ID-related syntax element is encoded in the image header for the LMCS or scaling list decoding tool; and in the stripe header, prior to syntax elements related to other decoding tools, at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe is encoded, wherein the ALF APS ID syntax element is parsed when enabled prior to at least one syntax element indicating whether the LMCS or scaling list decoding tool should be used for the stripe, wherein video data can be decoded from the bitstream using the syntax element.

[0040] In the eleventh aspect, a method for decoding a bit stream according to the tenth aspect is provided.

[0041] The tenth aspect, bitstream, can be represented by a signal carrying the bitstream. The signal can be carried on a transient (data on a wireless or wired data carrier, such as data downloaded or streamed via the Internet) or a non-transient (e.g., physical media, such as (Blu-ray) discs, memory, etc.) medium.

[0042] On the other hand, a method for encoding / decoding video data relative to a bitstream is provided, the method comprising: encoding / parsing in a picture header a syntax element indicating whether an LMCS or scaling list decoding tool can be used in the picture; and encoding / parsing in a slice header, prior to syntax elements related to other decoding tools, at least one syntax element indicating whether an LMCS or scaling list decoding tool should be used for the slice. Other decoding tools may be low-level decoding tools and do not contain an ALF. An adaptive loop filter APS ID syntax element may be encoded / parsed prior to at least one syntax element indicating whether an LMCS or scaling list decoding tool should be used for the slice (e.g., when enabled). Video data can be encoded into / decoded from the bitstream according to the syntax elements. The method may further comprise: encoding / decoding in the picture header at least one syntax element indicating whether an LMCS or scaling list decoding tool can be used in the picture (wherein when an LMCS or scaling list decoding tool is used in the picture). An apparatus may be configured to perform this method. A computer program may include instructions that, when executed, cause (e.g., by one or more processors) to perform the method.

[0043] Any feature of one aspect of the invention may be applied to other aspects of the invention in any suitable combination. In particular, a method aspect may be applied to an apparatus aspect, and vice versa.

[0044] Furthermore, features implemented in hardware can be implemented in software, and vice versa. Any references to software and hardware features in this document should be interpreted accordingly.

[0045] Any device feature described herein can also be provided as a method feature, and vice versa. As used herein, component-plus-functional features can be alternatively expressed in terms of their corresponding structure (such as a properly programmed processor and associated memory).

[0046] It should also be understood that specific combinations of the various features described and defined in any aspect of the invention may be implemented, provided, and / or used independently. Attached Figure Description

[0047] Now, let's refer to the attached diagram as an example, in which:

[0048] Figure 1 This is a diagram used to illustrate the coding structures used in HEVC and VVC;

[0049] Figure 2 This is a schematic block diagram illustrating a data communication system that can implement one or more embodiments of the present invention;

[0050] Figure 3 This is a block diagram illustrating components of a processing apparatus that can implement one or more embodiments of the present invention;

[0051] Figure 4 This is a flowchart illustrating the steps of an encoding method according to an embodiment of the present invention;

[0052] Figure 5 This is a flowchart illustrating the steps of a decoding method according to an embodiment of the present invention;

[0053] Figure 6 This illustrates the structure of a bitstream in an exemplary encoding system, VVC.

[0054] Figure 7 This demonstrates Luma Modelling Chroma Scaling (LMCS).

[0055] Figure 8 Showing the sub-tools of LMCS;

[0056] Figure 9This is a diagram illustrating a system including an encoder or decoder and a communication network according to an embodiment of the present invention;

[0057] Figure 10 This is a schematic block diagram of a computing device for implementing one or more embodiments of the present invention;

[0058] Figure 11 This is a diagram showing a webcam system; and

[0059] Figure 12 This is a diagram showing a smartphone. Detailed Implementation

[0060] Figure 1 This relates to the coding structure used in the High Efficiency Video Coding (HEVC) video standard. Video sequence 1 consists of a series of digital images i. Each such digital image is represented by one or more matrices. The matrix coefficients represent pixels.

[0061] The sequence of images 2 can be segmented into strips 3. In some cases, a strip can constitute the entire image. These strips are segmented into non-overlapping coding tree units (CTUs). A coding tree unit (CTU) is the basic processing unit of the High Efficiency Video Coding (HEVC) video standard and conceptually corresponds structurally to the macroblock unit used in several previous video standards. A CTU is sometimes also called a maximum coding unit (LCU). A CTU has luma and chroma component parts, each component part being called a coding tree block (CTB). These different color components are not... Figure 1 As shown in the image.

[0062] CTUs are typically 64 pixels x 64 pixels in size. Quadtree decomposition can be used to iteratively divide each CTU into smaller, variable-size coding units (CUs).

[0063] The coding unit is the basic coding element and consists of two seed units called the prediction unit (PU) and the transform unit (TU). The maximum size of the PU or TU is equal to the size of the CU. The prediction unit corresponds to the partition of the CU used for predicting pixel values. It is possible to partition the CU into various different partitions of the PU, as shown in 606, including partitions divided into 4 square PUs and two different partitions divided into 2 rectangular PUs. The transform unit is the basic unit for spatial transformation using DCT. The CU can be partitioned into TUs based on a quadtree representation 607.

[0064] Each stripe is embedded in a Network Abstraction Layer (NAL) unit. Additionally, the encoding parameters of the video sequence are stored in dedicated NAL units called parameter sets. In HEVC and H.264 / AVC, two types of parameter set NAL units are used: First, the Sequence Parameter Set (SPS) NAL unit, which collects all parameters that remain unchanged throughout the entire video sequence. Typically, it handles the encoding profile, video frame size, and other parameters. Second, the Picture Parameter Set (PPS) NAL unit, which includes parameters that can be changed from one picture (or frame) in the sequence to another picture (or frame). HEVC also includes the Video Parameter Set (VPS) NAL unit, which contains parameters describing the overall structure of the bitstream. VPS is a new type of parameter set defined in HEVC and is applied to all layers of the bitstream. A layer can contain multiple temporal sublayers, and all version 1 bitstreams are confined to a single layer. HEVC has certain layer extensions for scalability and multiple views, and these extensions will allow multiple layers with a backward-compatible version 1 base layer.

[0065] Figure 2 An example of a data communication system that can implement one or more embodiments of the present invention is illustrated. The data communication system includes a transmission device (in this case, server 201) operable to transmit data packets of a data stream to a receiving device (in this case, client terminal 202) via a data communication network 200. The data communication network 200 can be a wide area network (WAN) or a local area network (LAN). Such a network can be, for example, a wireless network (Wifi / 802.11a or b or g), an Ethernet network, an Internet network, or a hybrid network consisting of several different networks. In a particular embodiment of the invention, the data communication system can be a digital television broadcasting system, wherein server 201 sends the same data content to multiple clients.

[0066] The data stream 204 provided by server 201 may consist of multimedia data representing video and audio data. In some embodiments of the invention, the audio and video data streams may be captured by server 201 using a microphone and a camera, respectively. In some embodiments, the data stream may be stored on server 201 or received by server 201 from other data providers, or generated at server 201. Server 201 is provided with encoders for encoding the video and audio streams, particularly for providing compressed bitstreams for transmission, which are more compact representations of the data presented as input to the encoder.

[0067] To achieve a better ratio of data quality to data volume, video data can be compressed, for example, according to HEVC or H.264 / AVC formats.

[0068] Client 202 receives the transmitted bit stream and decodes the reconstructed bit stream to reproduce video images on a display device and reproduce audio data using a speaker.

[0069] Despite Figure 2 The examples consider streaming scenarios, but it will be appreciated that in some embodiments of the invention, media storage devices such as optical discs can be used for data communication between the encoder and decoder.

[0070] In one or more embodiments of the invention, video images are transmitted together with data representing compensation offsets of reconstructed pixels to be applied to the image, so as to provide filtered pixels in the final image.

[0071] Figure 3 A processing device 300 configured to implement at least one embodiment of the present invention is illustrated schematically. The processing device 300 may be a device such as a microcomputer, workstation, or lightweight portable device. The device 300 includes a communication bus 313 connected to:

[0072] The central processing unit 311, referred to as the CPU, is such as a microprocessor.

[0073] The read-only memory 306, designated as ROM, is used to store the computer program that implements the present invention;

[0074] Random access memory 312, represented as RAM, is used to store executable code of the methods according to embodiments of the present invention, and registers are suitable for recording variables and parameters required for implementing the method for encoding a digital image sequence and / or the method for decoding a bit stream according to embodiments of the present invention; and

[0075] The communication interface 302 is connected to the communication network 303, through which digital data to be processed is transmitted or received.

[0076] Optionally, the device 300 may also include the following components:

[0077] Data storage component 304, such as a hard disk, is used to store computer programs for implementing one or more embodiments of the present invention, as well as data used or generated during the implementation of one or more embodiments of the present invention;

[0078] Disk drive 305 for disk 306, the disk drive being adapted to read data from disk 306 or write data to disk;

[0079] Screen 309 is used to display data and / or serve as a graphical interface for user interaction via keyboard 310 or any other indicating device.

[0080] Device 300 can be connected to various peripheral devices such as digital camera 320 or microphone 308, each of which is connected to an input / output card (not shown) to provide multimedia data to device 300.

[0081] The communication bus provides communication and interoperability between various elements included in or connected to device 300. The representation of the bus is not limiting, and in particular, the central processing unit is operable to communicate instructions directly or by means of other elements of device 300 to any element of device 300.

[0082] Disk 306 may be replaced by any information medium such as a rewritable or non-rewritable compact disc (CD-ROM), ZIP disc, or memory card, and generally by an information storage component that can be read by a microcomputer or microprocessor. Disk 306 may be integrated into or not integrated into the device, may be portable, and is adapted to store one or more programs that execute to enable the implementation of the method for encoding digital image sequences and / or the method for decoding bit streams according to the present invention.

[0083] Executable code can be stored in read-only memory 306, hard disk 304, or removable digital media (such as, for example, disk 306 as described above). According to a variation, the executable code of the program can be received via interface 302 through communication network 303 to be stored in one of the storage components of device 300 (such as hard disk 304) before execution.

[0084] The central processing unit 311 is adapted to control and direct the execution of instructions or software code portions of one or more programs according to the invention, stored in one of the aforementioned storage components. Upon power-up, one or more programs stored in non-volatile memory (e.g., on hard disk 304 or in read-only memory 306) are transferred to random access memory 312 (which then contains executable code for one or more programs) and registers for storing variables and parameters necessary for implementing the invention.

[0085] In this embodiment, the device is a programmable device that implements the invention using software. However, alternatively, the invention can be implemented in hardware (e.g., in the form of an application-specific integrated circuit or ASIC).

[0086] Figure 4 A block diagram illustrating an encoder according to at least one embodiment of the present invention is shown. The encoder is represented by connected modules, each module being adapted to implement, for example, in the form of programming instructions executed by the CPU 311 of the device 300, at least one corresponding step of a method for encoding images in an image sequence according to one or more embodiments of the present invention.

[0087] Encoder 400 receives digital images i0 to i n The original sequence 401 is used as input. Each digital image is represented by a set of samples (called pixels).

[0088] After performing the encoding process, the encoder 400 outputs a bit stream 410. The bit stream 410 includes multiple encoding units or stripes, each strip including a strip header for transmitting the encoded values ​​of the encoding parameters used for strip encoding, and a strip body including encoded video data.

[0089] Module 402 will input digital images i0 to i n The image is divided into 401 pixel blocks. These blocks correspond to portions of the image and can have variable sizes (e.g., 4×4, 8×8, 16×16, 32×32, 64×64, 128×128 pixels, and several rectangular block sizes can also be considered). A coding mode is selected for each input block. Two families of coding modes are provided: a spatial prediction-based coding mode (intra-frame prediction) and a temporal prediction-based coding mode (inter-frame coding, merging, skipping). Possible coding modes were tested.

[0090] Module 403 implements intra-frame prediction processing, wherein the block to be encoded is predicted by means of predictors calculated based on the neighboring pixels of the given block. If intra-frame coding is selected, the selected intra-frame predictors and an indication of the difference between the given block and its predictors are encoded to provide residuals.

[0091] Temporal prediction is implemented by motion estimation module 404 and motion compensation module 405. First, a reference image is selected from reference image set 416, and motion estimation module 404 selects a portion of the reference image (also referred to as a reference region or image portion) that is closest to the given block to be encoded. Then, motion compensation module 405 uses the selected region to predict the block to be encoded. Motion compensation module 405 calculates the difference between the selected reference region and the given block (also referred to as the residual block). The selected reference region is indicated by a motion vector.

[0092] Therefore, in both cases (spatial and temporal predictions), the residuals are calculated by subtracting the predictions from the original blocks.

[0093] In intra-frame prediction implemented by module 403, the prediction direction is encoded. In temporal prediction, at least one motion vector is encoded. In inter-frame prediction implemented by modules 404, 405, 416, 418, and 417, at least one motion vector or data used to identify such a motion vector is encoded for temporal prediction.

[0094] If inter-frame prediction is selected, information related to motion vectors and residual blocks is encoded. To further reduce the bit rate, assuming the motion is homogeneous, the motion vectors are encoded by the difference relative to the motion vector predictors. The motion vector predictors in the set of motion information predictors are obtained from the motion vector field 418 by the motion vector prediction and encoding module 417.

[0095] The encoder 400 also includes a selection module 406, which selects the encoding mode by applying an encoding cost criterion (such as a rate-distortion criterion). To further reduce redundancy, a transform module 407 applies a transform (such as DCT) to the residual block, and then the obtained transform data is quantized by a quantization module 408 and entropy encoded by an entropy encoding module 409. Finally, the encoded residual block of the current block being encoded is inserted into the bit stream 410.

[0096] Encoder 400 also decodes the encoded image to produce a reference image for motion estimation of subsequent images. This allows the encoder and decoder receiving the bitstream to have the same reference frame. Inverse quantization module 411 performs inverse quantization of the quantized data, followed by inverse transform by inverse transform module 412. Inverse intra-frame prediction module 413 uses the prediction information to determine which predictor to use for a given block, and inverse motion compensation module 414 actually adds the residual obtained by module 412 to the reference region obtained from reference image set 416.

[0097] Then, module 415 applies post-filtering to filter the reconstructed pixel frames. In embodiments of the invention, a SAO loop filter is used, wherein a compensation offset is added to the pixel values ​​of the reconstructed pixels in the reconstructed image.

[0098] Figure 5 A block diagram of a decoder 60 according to an embodiment of the present invention is shown. The decoder 60 can be used to receive data from an encoder. The decoder is represented by connected modules, each module being adapted to implement corresponding steps of the method implemented by the decoder 60, for example, in the form of programming instructions to be executed by the CPU 311 of the device 300.

[0099] Decoder 60 receives bitstream 61 including coding units, each coding unit consisting of a header containing information related to the encoded parameters and a body containing the encoded video data. See below for reference. Figure 6 A more detailed description of the bitstream structure in VVC. For example, regarding... Figure 4 As described, for a given block, entropy encoding is performed on the encoded video data at a predetermined number of bits, and the index of the motion vector predictor is also encoded. The received encoded video data is entropy decoded by module 62. The residual data is then dequantized by module 63, and subsequently, an inverse transform is applied by module 64 to obtain the pixel values.

[0100] The pattern data used to indicate the encoding mode is also entropy decoded, and based on this pattern, intra-frame type decoding or inter-frame type decoding is performed on the encoded blocks of image data.

[0101] In intra-frame mode, the intra-frame inverse prediction module 65 determines the intra-frame predictor based on the intra-frame prediction mode specified in the bitstream.

[0102] If the mode is inter-frame, motion prediction information is extracted from the bitstream to find the reference region used by the encoder. The motion prediction information consists of a reference frame index and a motion vector residual. Motion vector predictors are added to the motion vector residual to obtain the motion vector by the motion vector decoding module 70.

[0103] Motion vector decoding module 70 applies motion vector decoding to each current block encoded by motion prediction. Once the index of the motion vector predictor for the current block has been obtained, the actual value of the motion vector associated with the current block can be decoded, and this actual value is used to apply inverse motion compensation via module 66. The reference image portion indicated by the decoded motion vectors is extracted from the reference image 68 to apply inverse motion compensation 66. The motion vector field data 71 is updated using the decoded motion vectors for subsequent inverse prediction of decoded motion vectors.

[0104] Finally, the decoded block is obtained. Post-filtering is applied by post-filter module 67. Decoder 60 finally provides the decoded video signal 69.

[0105] Figure 6 The organization of bitstreams in an exemplary encoding system VVC as described in JVET_P2001-VE is shown.

[0106] According to the VVC encoding system, bitstream 61 consists of an ordered sequence of syntax elements and encoded data. The syntax elements and encoded data are placed in Network Abstraction Layer (NAL) units 601-608. Different NAL unit types exist. The NAL provides the ability to encapsulate bitstreams into different protocols (such as RTP / IP, ISO Basic Media File Format, etc.). The NAL also provides a framework for mitigating packet loss.

[0107] The NAL unit is divided into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units contain the actual encoded video data. Non-VCL NAL units contain additional information. This additional information may be parameters required for decoding the encoded video data or supplementary data that enhances the usability of the decoded video data. NAL unit 606 corresponds to a stripe and constitutes a VCL NAL unit in the bitstream.

[0108] Different NAL units 601-605 correspond to different parameter sets; these NAL units are non-VCL NAL units. The Decoder Parameter Set (DPS) NAL unit 301 contains parameters that are constant for a given decoding process. The Video Parameter Set (VPS) NAL unit 602 contains parameters defined for the entire video and therefore the entire bitstream. DPS NAL units can define parameters that are more static than those in the VPS. In other words, the parameters in the DPS change less frequently than those in the VPS.

[0109] The Sequence Parameter Set (SPS) NAL unit 603 contains parameters defined for the video sequence. Specifically, the SPS NAL unit defines the sub-picture layout and associated parameters of the video sequence. Parameters associated with each sub-picture specify the coding constraints applied to the sub-picture. In particular, this includes flags indicating that temporal predictions between sub-pictures are limited to data from the same sub-picture. Another flag can enable or disable loop filters across sub-picture boundaries.

[0110] The Picture Parameter Set (PPS) NAL unit 604 contains parameters defined for a picture or group of pictures. The Adaptive Parameter Set (APS) NAL unit 605 contains parameters for a loop filter, which is typically an Adaptive Loop Filter (ALF) or a shaper model (or a Luminance Mapping with Chroma Scaling (LMCS) model) or a scaling matrix used at the stripe level.

[0111] The syntax for PPS, as proposed in the current version of VVC, includes syntax elements that specify the size of the image in units of luminance samples and the partitioning of each image into blocks and stripes.

[0112] The PPS contains syntactic elements that allow the location of stripes within a frame to be determined. Since subpicks form rectangular regions within a frame, the set of stripes, block portions, or blocks belonging to a subpicks can be determined based on the parameter set NAL unit. The PPS, as an APS, has an ID mechanism to limit the number of identical PPSs sent.

[0113] The main difference between PPS and image headers lies in their transmission. Unlike PHs, which are sent systematically for each image, PPSs are typically sent for groups of images. Therefore, PPSs contain parameters that can be constant for a number of images, unlike PHs.

[0114] The bitstream can also contain supplemental enhancement information (SEI) NAL units. Figure 6(Not shown in the text). The occurrence period of these parameter sets in the bitstream is variable. A VPS defined for the entire bitstream may appear only once in the bitstream. Conversely, an APS defined for a strip may appear once for each strip in each image. In practice, different stripes may depend on the same APS, and therefore there are usually fewer APSs than there are stripes in each image. In particular, the APS is defined in the image header. However, ALF APSs can be refined in the strip header.

[0115] The Access Unit Delimiter (AUD) of NAL Unit 607 separates two access units. An access unit is a collection of NAL units that can include one or more encoded pictures with the same decoding timestamp. This optional NAL unit contains only one syntax element from the current VVC specification: `pic_type`, which indicates the `slice_type` value for all slices of the encoded picture in the AU. If `pic_type` is set to 0, the AU contains only intra slices. If it is 1, it contains both P and I slices. If it is 2, it contains B, P, or intra slices.

[0116] This NAL unit contains only one syntax element, pic-type.

[0117] Table 1 Syntactic AUD

[0118]

[0119] In JVET-P2001-VE, pic-type is defined as follows:

[0120] "The slice_type value for all slices of the encoded picture in the AU containing the AU delimiter NAL unit is a member of the set listed in Table 2 for a given pic_type value. In a bitstream conforming to this version of the specification, the value of pic_type should be equal to 0, 1, or 2. Other values ​​of pic_type are reserved for future use by ITU-T|ISO / IEC. Decoders conforming to this version of the specification will ignore the reserved values ​​of pic_type."

[0121] Table 2 Explanation of pic-type

[0122]

[0123] PH NAL unit 608 is the image header NAL unit, which groups common parameters of a set of coded image stripes. The image may refer to one or more APSs to indicate the AFL parameters, shaper model, and scaling matrix used by the image stripes.

[0124] VCL NAL units 606 each contain stripes. A stripe can correspond to an entire image or a sub-image, a single block, or a fragment of multiple blocks or blocks. For example, Figure 3 The stripe contains several blocks 620. The stripe consists of a stripe header 610 and a raw byte sequence payload RBSP 611, which contains encoded pixel data encoded as encoded blocks 640.

[0125] The syntax for PPS, as proposed in the current version of VVC, includes syntax elements that specify the size of the image in units of luminance samples and the partitioning of each image in units of blocks and stripes.

[0126] PPS contains syntactic elements that allow the location of stripes within a frame to be determined. Since sub-pictures form rectangular regions within a frame, the set of stripes, block portions, or blocks belonging to a sub-picture can be determined from the parameter set NAL units.

[0127] NAL unit strips

[0128] The NAL unit stripe layer contains stripe headers and stripe data, as shown in Table 3.

[0129] Table 3 Strip Layer Syntax

[0130]

[0131] APS

[0132] The Adaptive Parameter Set (APS) NAL unit 605 is defined in Table 4, which shows the syntactic elements.

[0133] As depicted in Table 4, there are three possible types of APS given by the aps_params_type syntax element:

[0134] - ALF_AP: Used for ALF parameters

[0135] - LMCS_APS: Used for LMCS parameters

[0136] - SCALLING_APS: Parameters used for scaling the list.

[0137] Table 4 Adaptive Parameter Set Syntax

[0138]

[0139] The three types of APS parameters will be discussed in turn below.

[0140] ALF APS

[0141] The ALF parameters are described in the Adaptive Loop Filter data syntax elements (Table 5). First, two flags are dedicated to specifying whether an ALF filter is sent for luma and / or chroma. If the luma filter flag is enabled, another flag is decoded to determine whether a clipping value (alf_luma_clip_flag) is signaled. Then, the number of signaled filters is decoded using the alf_luma_num_filters_signalled_minus1 syntax element. If necessary, the syntax element representing the ALF coefficient increment “alf_luma_coeff_delta_idx” is decoded for each enabled filter. Then, the absolute value and sign of each coefficient for each filter are decoded.

[0142] If alf_luma_clip_flag is enabled, the clipping index of each coefficient of each enabled filter is decoded.

[0143] In the same way, decode the ALF chroma coefficients when needed.

[0144] Table 5 Adaptive Loop Filter Data Syntax

[0145]

[0146]

[0147] LMCS syntax elements used for both luminance mapping and chrominance scaling.

[0148] Table 6 below shows all LMCS syntax elements (LMCS_APS) encoded in the Adaptive Parameter Set (APS) syntax structure when the aps_params_type parameter is set to 1. Up to four LMCS APSs can be used in an encoded video sequence; however, only a single LMCS APS can be used for a given image.

[0149] These parameters are used to construct the forward and backward mapping functions for luminance and the scaling function for chrominance.

[0150] Table 6 Luminance Maps with Chroma Scaling Data Syntax

[0151]

[0152] Zoom List AP

[0153] The scaling list provides the possibility to update the quantization matrix used for quantization. In VVC, this scaling matrix is ​​in the APS as described in the scaling list data syntax elements (Table 7). This syntax specifies whether the scaling matrix is ​​used for the LFNST (Low Frequency Inseparable Transform) tool based on the flag scaling_matrix_for_lfnst_disabled_flag. Then, the syntax elements required to construct the scaling matrix are decoded (scaling_list_copy_mode_flag, scaling_list_pred_mode_flag, scaling_list_pred_id_delta, scaling_list_dc_coef, scaling_list_delta_coef).

[0154] Table 7 Scaling List Data Syntax

[0155]

[0156] Image header

[0157] Image headers are sent at the beginning of each image. The image header table syntax currently contains 82 syntactic elements and approximately 60 test conditions or loops. This is significantly larger than the previous headers in earlier drafts of the standard. A complete description of all these parameters can be found in JVET_P2001-VE. Table 8 shows these parameters in the current image header decoding syntax. In this simplified version of the table, some syntactic elements have been grouped for ease of reading.

[0158] The relevant syntactic elements that can be decoded include:

[0159] - Whether to use this image / reference frame

[0160] - Output Frame

[0161] - Use sub-images (if needed)

[0162] - List of reference images (if needed)

[0163] - Color plane (if needed)

[0164] - Partition update (if the overwrite flag is enabled)

[0165] - Incremental QP parameters (if needed)

[0166] - Motion information parameters (if needed)

[0167] - ALF parameters (if needed)

[0168] - SAO parameters (if needed)

[0169] - Quantization parameters (if needed)

[0170] - LMCS parameters (if needed)

[0171] - Scaling list parameters (if needed)

[0172] - Image header extension (if needed)

[0173] The first three flags, which have a fixed length, are non_reference_picture_flag, gdr_pic_flag, and no_output_of_prior_pics_flag, which provide information related to the characteristics of the images within the bitstream.

[0174] Then, if gdr_pic_flag is enabled, recovery_poc_cnt is decoded.

[0175] The PPS parameter "PS_parameters()" sets the PPS ID and some other information (if needed). It contains three syntactic elements:

[0176] - ph_pic_parameter_set_id

[0177] - ph_poc_msb_present_flag

[0178] - poc_msb_val

[0179] The subpic parameter `Subpic_parameter()` is enabled when the subpick ID is enabled at the SPS and if signaling is disabled. It also contains information about virtual boundaries. Eight syntax elements are defined for the subpic parameter:

[0180] - ph_subpic_id_signalling_present_flag

[0181] - ph_subpic_id_len_minus1

[0182] - ph_subpic_id

[0183] - ph_loop_filter_across_virtual_boundaries_disabled_present_flag

[0184] - ph_num_ver_virtual_boundaries

[0185] - ph_virtual_boundaries_pos_x

[0186] - ph_num_hor_virtual_boundaries

[0187] - ph_virtual_boundaries_pos_y

[0188] Then, if the sequence contains a separate color plane, the color plane id "colour_plane_id" is decoded, followed by decoding pic_output_flag (if it exists).

[0189] Then, the parameters for the reference picture list are decoded into "reference_picture_list_parameters()", which contains the following syntax elements:

[0190] - pic_rpl_present_flag

[0191] - pic_rpl_sps_flag

[0192] - pic_rpl_idx

[0193] - pic_poc_lsb_lt

[0194] - pic_delta_poc_msb_present_flag

[0195] - pic_delta_poc_msb_cycle_lt

[0196] If needed, another set of syntax elements can also be decoded related to the reference picture list structure "ref_pic_list_struct", which contains eight other syntax elements.

[0197] If necessary, the set of partitioning parameters "partitioning_parameters()" is decoded, and it contains the following 13 syntactic elements:

[0198] - partition_constraints_override_flag

[0199] - pic_log2_diff_min_qt_min_cb_intra_slice_luma

[0200] - pic_log2_diff_min_qt_min_cb_inter_slice

[0201] - pic_max_mtt_hierarchy_depth_inter_slice

[0202] - pic_max_mtt_hierarchy_depth_intra_slice_luma

[0203] - pic_log2_diff_max_bt_min_qt_intra_slice_luma

[0204] - pic_log2_diff_max_tt_min_qt_intra_slice_luma

[0205] - pic_log2_diff_max_bt_min_qt_inter_slice

[0206] - pic_log2_diff_max_tt_min_qt_inter_slice

[0207] - pic_log2_diff_min_qt_min_cb_intra_slice_chroma

[0208] - pic_max_mtt_hierarchy_depth_intra_slice_chroma

[0209] - pic_log2_diff_max_bt_min_qt_intra_slice_chroma

[0210] - pic_log2_diff_max_tt_min_qt_intra_slice_chroma

[0211] Following these partitioning parameters, the four incremental QP syntax elements "Delta_QP_Parameters()" can be decoded if needed:

[0212] - pic_cu_qp_delta_subdiv_intra_slice

[0213] - pic_cu_qp_delta_subdiv_inter_slice

[0214] - pic_cu_chroma_qp_offset_subdiv_intra_slice

[0215] - pic_cu_chroma_qp_offset_subdiv_inter_slice

[0216] Then, if necessary, decode pic_joint_cbcr_sign_flag, followed by decoding the set of three SAO syntax elements "SAO_parameters()":

[0217] - pic_sao_enabled_present_flag

[0218] - pic_sao_luma_enabled_flag

[0219] - pic_sao_chroma_enabled_flag

[0220] Then, if ALF is enabled at the SPS level, the set of ALF APS id syntax elements is decoded.

[0221] First, decode `pic_alf_enabled_present_flag` to determine if `pic_alf_enabled_flag` should be decoded. If `pic_alf_enabled_flag` is enabled, ALF is enabled for all stripes of the current image.

[0222] If ALF is enabled, the amount of ALF APSid for luminance is decoded using the pic_num_alf_aps_ids_luma syntax element. For each APS id, the luminance-specific APS id value is decoded as "pic_alf_aps_id_luma".

[0223] For chroma, the decoding syntax element `pic_alf_chroma_idc` determines whether ALF is enabled for chroma, Cr only, or Cb only. If enabled, the `pic_alf_aps_id_chroma` syntax element is used to decode the APS id value for chroma.

[0224] Following the set of ALF APS id parameters, the quantization parameters in the image header are decoded if necessary.

[0225] - pic_dep_quant_enabled_flag

[0226] - sign_data_hiding_enabled_flag

[0227] Then, if necessary, the set of three deblocking filter syntax elements "deblocking_filter_parameters()" is decoded:

[0228] - pic_deblocking_filter_override_present_flag

[0229] - pic_deblocking_filter_override_flag

[0230] - pic_deblocking_filter_disabled_flag

[0231] - pic_beta_offset_div2

[0232] - pic_tc_offset_div2

[0233] After enabling LMCS at SPS, the set of LMCS APS ID syntax elements is decoded. First, `pic_lmcs_enabled_flag` is decoded to determine if LMCS is enabled for the current image. If LMCS is enabled, the ID value is decoded as `pic_lmcs_aps_id`. For chroma, only `pic_chroma_residual_scale_flag` is decoded to enable or disable the chroma-specific approach.

[0234] Then, if the scaling list is enabled at the SPS level, the set of scaling list APS IDs is decoded. `pic_scaling_list_present_flag` is decoded to determine whether a scaling matrix is ​​enabled for the current image. The value of the APS ID, `pic_scaling_list_aps_id`, is then decoded. When the scaling list is enabled at the sequence level (i.e., the SPS level) (`sps_scaling_list_enabled_flag` equals 1), and when the scaling list is enabled at the image header level (`pic_scaling_list_present_flag` equals 1), the flag `slice_scaling_list_present_flag` indicating whether the scaling list is enabled for the current slice is extracted from the bitstream in the slice header.

[0235] Finally, if necessary, the extended syntax elements of the image header are decoded.

[0236] Table 8. Headers of some images

[0237]

[0238]

[0239] strip head

[0240] A stripe header is sent at the beginning of each stripe. The stripe header table syntax currently contains 57 syntactic elements. This is significantly larger than previous stripe headers in earlier versions of the standard. A complete description of all stripe header parameters can be found in JVET_P2001-VE. Table 9 shows these parameters in the current picture header decoding syntax. In this simplified version of the table, some syntactic elements have been grouped to make the table more readable.

[0241] Table 9. Partial Strip Heads

[0242]

[0243]

[0244] First, decode slice_pic_order_cnt_lsb to determine the POC of the current slice.

[0245] Then, if necessary, `slice_subpic_id` is decoded to determine the subpick ID of the current slice. Then, `slice_address` is decoded to determine the address of the current slice. If the number of tiles in the current image is greater than 1, then `num_tiles_in_slice_minus1` is decoded.

[0246] Then decode slice_type.

[0247] Decode the reference image list parameter set; these are similar to the parameter set in the image header.

[0248] When the stripe type is not intra-frame, and if necessary, `cabac_init_flag` and / or `collocated_from_lo_flag` and `collocated_ref_idx` are decoded. These data relate to the juxtaposed CABAC encoding and motion vectors.

[0249] Similarly, when the stripe type is not intra-frame, the parameters of the weighted prediction pred_weight_table() are decoded.

[0250] In slice_quantization_parameters(), slice_qp_delta is systematically decoded before other parameters of the QP offset if necessary.

[0251] slice_cb_qp_offset, slice_cr_qp_offset, slice_joint_cbcr_qp_offset, cu_chroma_qp_offset_enabled_flag.

[0252] The SAO enable flags are decoded for both luminance and chroma: slice_sao_luma_flag and slice_sao_chroma_flag (if indicated by a signal in the image header).

[0253] Then, if necessary, the APS ALF ID is decoded, which has similar data constraints as in the image header.

[0254] Then, the deblocking filter parameters are decoded before the other data (slice_deblocking_filter_parameters()).

[0255] As in the image header, the ALF APS ID is set at the end of the strip header.

[0256] APS in the image header

[0257] As depicted in Table 8, the APS ID information for the three tools ALF, LMCS, and zoom list is located at the end of the image header syntax element.

[0258] Streaming applications

[0259] Some streaming applications extract only certain portions of the bitstream. These extractions can be spatial (as sub-pictures) or temporal (sub-sections of a video sequence). These extracted portions can then be merged with other bitstreams. Others reduce the frame rate by extracting only certain frames. Typically, the primary goal of these streaming applications is to produce the highest quality for the end user using the maximum allowed bandwidth.

[0260] In VVC, to reduce frame rates, APS ID numbers are restricted so that new APS ID numbers for frames cannot be used for frames in higher time layers. However, for streaming applications that extract portions of the bitstream, APS IDs need to be tracked to determine which APSs should be retained for sub-portions of the bitstream, because frames (due to IRAP) do not reset APS ID numbers.

[0261] LMCS (Luminance Map with Chroma Scaling)

[0262] Luminance Mapping with Chroma Scaling (LMCS) is a sample value transformation method applied to blocks before applying loop filters in video decoders such as VVC.

[0263] LMCS can be divided into two sub-tools. The first sub-tool is applied to the luma block, and the second sub-tool is applied to the chroma block, as described below:

[0264] 1) The first sub-tool is an intra-loop mapping of the luminance component based on an adaptive piecewise linear model. Intra-loop mapping of the luminance component adjusts the dynamic range of the input signal by redistributing codewords across the dynamic range to improve compression efficiency. The luminance mapping utilizes a forward mapping function in the "mapping domain" and a corresponding inverse mapping function returning to the "input domain".

[0265] 2) The second sub-tool relates to the chromaticity component of the applied luminance-dependent chromaticity residual scaling. Chromaticity residual scaling is designed to compensate for the interaction between the luminance signal and its corresponding chromaticity signal. Chromaticity residual scaling depends on the average of the adjacent luminance samples reconstructed from the upper and / or left sides of the current block.

[0266] Like most other tools in video encoders (such as VVC), LMCS can be enabled / disabled at the sequence level using the SPS flag. Chroma residual scaling is also signaled at the picture level. If luma mapping is enabled at the picture level, an additional flag is signaled to indicate whether luma-dependent chroma residual scaling is enabled. When luma mapping is not used at the SPS level, luma-dependent chroma residual scaling is completely disabled for all pictures with a reference SPS. Additionally, luma-dependent chroma residual scaling is always disabled for chroma blocks smaller than or equal to 4. When LMCS is enabled at the SPS level, the flag `pic_lmcs_enabled_flag` is decoded in the picture header to determine whether LMCS is enabled for the current picture. When LMCS is enabled at the picture level (`pic_lmcs_enabled_flag` equals 1), another flag, `slice_lmcs_enabled_flag`, is decoded for each slice. This flag indicates whether LMCS for the current slice is enabled using the parameters decoded in the picture header. There is no APS ID information related to LMCS in the slice header. As a result, only information related to enabling or disabling LMCS exists.

[0267] Figure 7 The principle of LMCS as described above for the luminance mapping sub-tool is shown. Figure 7The shadow block in the diagram is a new LMCS function block that includes forward and reverse mapping of the luminance signal. It's important to note that when using LMCS, some decoding operations are applied in the "mapping domain." These operations are handled by... Figure 7 The dashed blocks in the image represent these steps. They typically correspond to the inverse quantization, inverse transform, intra-frame lumen prediction, and reconstruction steps (which involve adding the lumen prediction to the lumen residual). Conversely, Figure 7 The solid-line blocks in the diagram indicate the locations in the original (i.e., unmapped) domain where decoding processing is applied, and this includes loop filtering such as deblocking, ALF and SAO, motion compensation prediction, and storage of the decoded image as a reference image (DPB).

[0268] Figure 8 Showing with Figure 7 Similar diagram, but this time it's for the chroma scaling sub-tool of the LMCS tool. Figure 8 The shadow blocks in the diagram are new LMCS function blocks that include luma-dependent chroma scaling processing. However, there are some important differences in chroma compared to the luma case. Here, for chroma samples, inverse quantization and inverse transform, represented by the blocks in the dashed lines, are performed only in the "mapping domain." All other steps, including intra-frame chroma prediction, motion compensation, and loop filtering, are performed in the original domain. Figure 8 As shown, for brightness mapping, there is only scaling processing, and there is no forward or reverse processing.

[0269] Luminance mapping using a piecewise linear model

[0270] The luminance mapping sub-tool uses a piecewise linear model. This means that the piecewise linear model divides the dynamic range of the input signal into 16 equal sub-ranges, and for each sub-range, its linear mapping parameters are represented by the number of codewords assigned to that range.

[0271] Semantics of brightness mapping

[0272] The syntax element lmcs_min_bin_idx specifies the minimum bin (range) index used in the build process of a Luminance Map with Chroma Scaling (LMCS). The value of lmcs_min_bin_idx should be in the range of 0 to 15 (inclusive).

[0273] The syntactic element `lmcs_delta_max_bin_idx` specifies the increment between 15 and the maximum bin index `LmcsMaxBinIdx` used in the build process of a luma map with chroma scaling. The value of `lmcs_delta_max_bin_idx` should be in the range of 0 to 15 (inclusive). The value of `LmcsMaxBinIdx` is set to be equal to 15 - `lmcs_delta_max_bin_idx`. The value of `LmcsMaxBinIdx` should be greater than or equal to `lmcs_min_bin_idx`.

[0274] The syntax element lmcs_delta_cw_prec_minus1 plus 1 specifies the number of bits used to represent the syntax lmcs_delta_abs_cw[i].

[0275] The syntactic element lmcs_delta_abs_cw[i] specifies the absolute increment codeword value of the i-th bin.

[0276] The syntactic element lmcs_delta_sign_cw_flag[i] specifies the sign of the variable lmcsDeltaCW[i]. If lmcs_delta_sign_cw_flag[i] does not exist, it is inferred to be equal to 0.

[0277] LMCS intermediate variable calculation for luminance mapping

[0278] To apply forward and reverse brightness mapping, some intermediate variables and data arrays are required.

[0279] First, derive the variable OrgCW as follows:

[0280] OrgCW=(1< <BitDepth) / 16

[0281] Then, the variable lmcsDeltaCW[i] (where i = lmcs_min_bin_idx…LmcsMaxBinIdx) is calculated as follows:

[0282] lmcsDeltaCW[i]=(1-2 lmcs_delta_sign_cw_flag[i]) lmcs_delta_abs_cw[i]

[0283] The new variable lmcsCW[i] is exported as follows:

[0284] For i=0…lmcs_min_bin_idx-1, lmcsCW[i] is set to equal to 0.

[0285] For i = lmcs_min_bin_idx … LmcsMaxBinIdx, the following applies:

[0286] lmcsCW[i]=OrgCW+lmcsDeltaCW[i]

[0287] The value of lmcsCW[i] should be in the range of (OrgCW>>3) to (OrgCW<<3-1) (inclusive).

[0288] For i = LmcsMaxBinIdx + 1…15, lmcsCW[i] is set to equal to 0.

[0289] The variable InputPivot[i] (where i = 0…16) is derived as follows:

[0290] InputPivot[i]=i OrgCW

[0291] The variables LmcsPivot[i] (where i = 0…16), ScaleCoeff[i], and InvScaleCoeff[i] (where i = 0…15) are calculated as follows:

[0292] LmcsPivot[0] = 0;

[0293] for( i = 0; i <= 15; i++ ) {

[0294] LmcsPivot[ i + 1 ] = LmcsPivot[ i ] + lmcsCW[ i ]

[0295] ScaleCoeff[ i ] = ( lmcsCW[ i ] (1 << 11 ) + ( 1 << ( Log2(OrgCW ) - 1 ) ) ) >> ( Log2( OrgCW ) )

[0296] if ( lmcsCW[ i ] == 0 )

[0297] InvScaleCoeff[i] = 0

[0298] else

[0299] InvScaleCoeff[i] = OrgCW (1 << 11) / lmcsCW[i]

[0300] Forward luminance mapping

[0301] As Figure 7 shown, when the LMCS is applied to luminance, luminance remapped samples called predMapSamples[i][j] are obtained from the prediction samples predSamples[i][j].

[0302] predMapSamples[i][j] is calculated as follows:

[0303] First, an index idxY is calculated from the prediction sample predSamples[i][j] at the position (i, j).

[0304] idxY = predSamples[i][j] >> Log2(OrgCW)

[0305] Then, predMapSamples[i][j] is derived as follows using the intermediate variables idxY, LmcsPivot[idxY], and InputPivot[idxY] with part 0:

[0306] predMapSamples[ i ][ j ] = LmcsPivot[ idxY ]

[0307] + ( ScaleCoeff[ idxY ] * ( predSamples[ i ][ j ] - InputPivot[ idxY ]) + ( 1 << 10 ) ) >> 11

[0308] Luminance reconstruction samples

[0309] Reconstruction processing is obtained from the predicted luminance samples predMapSample[i][j] and the residual luminance samples resiSamples[i][j].

[0310] The reconstructed luminance picture samples recSamples[i][j] are simply obtained by adding predMapSample[i][j] to resiSamplei[i][j] as follows:

[0311] recSamples[i][j] = Clip1(predMapSamples[i][j] + resiSamples[i][j]])

[0312] In the above relationship, the Clip 1 function is a clipping function to ensure that the reconstructed samples are between 0 and 1 << BitDepth - 1.

[0313] Inverse brightness mapping

[0314] When the application is based on Figure 7 During the inverse brightness mapping, the following operations are applied to each sample recSample[i][j] of the current block being processed:

[0315] First, the index idxY is calculated from the reconstructed sample recSamples[i][j] at position (i, j).

[0316] idxY=recSamples[i][j]>>Log2(OrgCW)

[0317] The inverse mapping of the luminance sample invLumaSample[i][j] is based on the following derivation:

[0318] invLumaSample[i][j] =

[0319] InputPivot[idxYInv] + (InvScaleCoeff[idxYInv]

[0320] ( recSample[i][j] - LmcsPivot[ idxYInv ] ) + ( 1 << 10 ) ) >> 11

[0321] Then, a cropping operation is performed to obtain the final sample:

[0322] finalSample[i][j]=Clip1(invLumaSample[i][j])

[0323] Chroma scaling

[0324] LMCS semantics for chroma scaling

[0325] The syntactic element lmcs_delta_abs_crs in Table 6 specifies the absolute codeword value of the variable lmcsDeltaCrs. The value of lmcs_delta_abs_crs should be in the range of 0 to 7 (inclusive). If it does not exist, lmcs_delta_abs_crs is inferred to be equal to 0.

[0326] The syntactic element lmcs_delta_sign_crs_flag specifies the sign of the variable lmcsDeltaCrs. If it does not exist, it is inferred that lmcs_delta_sign_crs_flag is equal to 0.

[0327] LMCS intermediate variable calculation for chroma scaling

[0328] In order to apply chroma scaling, some intermediate variables are needed.

[0329] The variable lmcsDeltaCrs is exported as follows:

[0330] lmcsDeltaCrs=(1-2 lmcs_delta_sign_crs_flag) lmcs_delta_abs_crs

[0331] The variable ChromaScaleCoeff[i] (where i = 0…15) is derived as follows:

[0332] if ( lmcsCW[ i ] == 0 )

[0333] ChromaScaleCoeff[ i ] = ( 1 << 11 )

[0334] else

[0335] ChromaScaleCoeff[i] = OrgCW ( 1 << 11 ) / ( lmcsCW[ i ] +lmcsDeltaCrs )

[0336] Chroma scaling

[0337] In the first step, the variable invAvgLuma is derived to calculate the average luminance value of the reconstructed luminance samples surrounding the current corresponding chroma block. The average luminance is calculated from the left and upper luminance blocks surrounding the corresponding chroma block.

[0338] If no samples are available, the variable invAvgLuma is set as follows:

[0339] invAvgLuma=1<<(BitDepth-1)

[0340] Based on the intermediate array LmcsPivot[] with a subset of zeros, the variable idxYInv is then derived as follows:

[0341] For (idxYInv = lmcs_min_bin_idx; idxYInv <= LmcsMaxBinIdx; idxYInv++) {

[0342] if(invAvgLuma < LmcsPivot [ idxYInv + 1 ] ) break

[0343] }

[0344] IdxYInv = Min( idxYInv, 15 )

[0345] The variable varScale is exported as follows:

[0346] varScale=ChromaScaleCoeff[idxYInv]

[0347] When applying a transformation to the current chroma block, the reconstructed chroma image sample array recSamples is exported as follows:

[0348] recSamples[i ][j ] = Clip1( predSamples[ i ][ j ] +

[0349] Sign( resiSamples[ i ][ j ] ) ( (Abs( resiSamples[ i ][ j ] ) varScale + ( 1 << 10 ) ) >> 11 ) )

[0350] If no transformation has been applied to the current block, the following applies:

[0351] recSamples[i][j]=Clip1(predSamples[i][j])

[0352] Encoder considerations

[0353] The basic principle of the LMCS encoder is to first allocate more codewords to ranges with a dynamic range that have a lower than average variance. In an alternative concept, the primary goal of LMCS is to allocate fewer codewords to dynamic range ranges with a higher than average variance. In this way, smooth regions of an image will be encoded with more codewords than average, and vice versa.

[0354] All parameters of the LMCS tool stored in the APS are determined on the encoder side (see Table 6). The LMCS encoder algorithm is based on the evaluation of local brightness variance, and the determination of the LMCS parameters is optimized according to the above basic principles. Optimization is then performed to obtain the optimal PSNR metric for the final reconstructed sample of a given block.

[0355] As will be discussed below, parsing complexity can be reduced by setting the APS ID information at the beginning of the strip / image header. This allows the parsing of APS ID-related syntax elements to precede the parsing of syntax elements related to the decoding tool.

[0356] Image header

[0357] The following description provides details or alternatives to the table below for the syntactic elements used in the image header. Table 10 shows an example modification to the image header (see Table 8 above), where the APS ID information for LMCS, ALF, and scaling lists is set close to the beginning of the image header to address parsing issues in the prior art. This change is indicated by strikethrough and underlined text. Using this table of syntactic elements, APS ID tracking has lower complexity.

[0358] Table 10 shows the partial image header of the modified APS ID information.

[0359]

[0360]

[0361]

[0362] APS ID-related syntactic elements may include one or more of the following:

[0363] - APS ID LMCS

[0364] - APS ID scaling list

[0365] ○ The APS scaling list contains information about the scaling matrix used to enable the LNST method.

[0366] - APS ID ALF List

[0367] ○ The APS ALF list contains information related to signaling the clipping value.

[0368] In one example, the APS ID-related syntax element is set at or near the beginning of the image header.

[0369] The advantage of this feature is that it reduces the complexity of streaming applications that need to track APS ID usage to remove unused APS NAL units.

[0370] As mentioned above, the word "begin" does not necessarily mean "the very beginning," because some syntactic elements may precede the syntactic elements related to APS ID. These will be discussed in turn below.

[0371] The set of APS ID-related syntax elements is set after one or more of the following:

[0372] - Syntactic elements useful for streaming applications that require parsing bitstreams to cut, split, or extract portions of a video sequence without parsing tool parameters.

[0373] - A syntactic element indicating that the current image has never been used as a reference image (e.g., non_reference_picture_flag), which, when equal to 1, indicates that the image has never been used as a reference image. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID LMCS syntactic elements.

[0374] ○ A syntax element indicating that the current image is a progressively decoded refresh image (e.g., the gdr_pic_flag syntax), which, when equal to 1, indicates that the current image is a GDR image. This is because this type of image means that the bitstream is independent of previous images. This is important information for many streaming applications. For the same reason, if gdr_pic_flag is enabled, the set of APS IDLMCS syntax elements should follow the decoded recovery_poc_cnt syntax elements.

[0375] - A syntactic element indicating changes to previously decoded pictures in the decoded picture buffer (e.g., no_output_of_prior_pics_flag) that affect the decoded picture buffer (DBP). This is important information for many applications.

[0376] - ph_pic_parameter_set_id and / or other syntactic elements related to PPS information. In fact, PPSid is required for many applications.

[0377] - Syntactic elements related to sub-image information. In fact, this information is important for bitstream extraction in some streaming applications.

[0378] - Image type information sent in the image header. In fact, image type is important information for many applications.

[0379] In a particularly advantageous example, the set of APS ID-related syntax elements is set after the syntax elements at the beginning of the image header (these syntax elements have fixed-length codewords without conditions for their decoding). The fixed-length codewords correspond to the syntax elements in the syntax element table that have descriptors u(N), where N is an integer value.

[0380] Compared to the current VVC specification, it corresponds to the non_reference_picture_flag, gdr_pic_flag, and no_output_of_prior_pics_flag flags.

[0381] This approach has the advantage of being ideal for streaming applications that can easily bypass these initial codewords (because the number of bits is always the same at the beginning of the image header) and directly access the useful information for the corresponding application. Therefore, the decoder can directly parse the APS ID-related syntax elements from the header, as they are always in the same bit position within the header.

[0382] In a variant that reduces parsing complexity, the set of APS ID-related syntax elements is set at the beginning of the image header after a syntax element that does not require one or more values ​​from another header. Parsing such syntax elements does not require additional variables for their parsing, thus reducing parsing complexity.

[0383] In one variant, the set of APS ID scaling list syntax elements is set at the beginning of the image header.

[0384] The advantage of this variant is that it reduces the complexity of streaming applications that require tracking APS ID usage to remove unused APS NAL units.

[0385] It should be understood that the above-mentioned types of APS ID-related syntactic elements can be processed individually or in combination. The following discusses a particularly advantageous combination of non-exhaustive lists.

[0386] The APS ID LMCS syntax element set and the APS ID zoom list syntax element set are set at or near the beginning of the image header, and optionally after useful information for streaming applications.

[0387] The APS ID LMCS syntax element set and the APS ID ALF syntax element set are set at the beginning of the image header and optionally after useful information for streaming applications.

[0388] The APS ID LMCS syntax element set, the APS ID zoom list syntax element set, and the APS ID ALF syntax element set are set at or near the beginning of the image header, and optionally after useful information for streaming applications.

[0389] In one embodiment, APS ID-related syntax elements (APS LMCS, APS scaling list, and / or APS ALF) are set before low-level tool parameters.

[0390] This reduces complexity for some streaming applications that need to track different APS IDs for individual images. In fact, the information relevant to the low-level tools is needed for parsing and decoding striped data, but not for applications that only extract the bitstream without decoding. An example of a low-level tool is provided below:

[0391] - A set of deblocking filter parameters; these parameters are used for the deblocking filters required only for striped data decoding.

[0392] - A set of quantization parameters; these parameters are required for strip data parsing and decoding.

[0393] - SAO parameter set; these parameters are required for striped data parsing and decoding.

[0394] - pic_joint_cbcr_sign_flag flags; these parameters are required for striped data parsing and decoding.

[0395] - A set of motion parameters; these parameters are required for strip data parsing and decoding.

[0396] - A set of QP offset parameters; these parameters are required for striped data parsing and decoding.

[0397] - A set of partitioning parameters; these parameters are required for striped data parsing and decoding.

[0398] - Reference image list parameter set; these parameters are required for strip data parsing and decoding.

[0399] - The pic_output_fl flag; these parameters are required for striped data decoding.

[0400] - colour_plane_id flags; these parameters are required for stripe data parsing and decoding.

[0401] It should be understood that the above-mentioned types of APS ID-related syntactic elements can be processed individually or in combination. The following discusses a particularly advantageous combination of non-exhaustive lists.

[0402] The APS ID LMCS syntax element set and the APS ID scaling list syntax element set are set before the low-level utility syntax elements.

[0403] The APS ID LMCS syntax element set and the APS ID ALF syntax element set are set before the low-level tool syntax elements.

[0404] The APS ID LMCS syntax element set, the APS ID scaling list syntax element set, and the APS ID ALF syntax element set are set before the low-level utility syntax elements.

[0405] Resynchronize images

[0406] Within a bitstream, there exist resynchronization frames or stripes. These frames allow the bitstream to be read without considering previously decoded information. For example, a Clean Random Access Picture (CRA picture) does not reference any other picture besides itself for inter-frame prediction during its decoding process. This means that the stripe of this picture is intra-frame or IBC. In VVC, a CRA picture is an IRAP (Intra-Frame Random Access Point) picture, where each VCL NAL unit has a nal unit_type equal to CRA_NUT.

[0407] In VVC, these resynchronization frames can be IRAP images or GDR images.

[0408] However, IRAP or GDR images, or IRAP stripes used for some configurations, may have an APS ID in their image header derived from a previously decoded APS. Therefore, the APS ID should be tracked for true random access points or when splitting the bitstream.

[0409] One way to improve this is to reset all APS IDs and / or disallow APS from other decoded APS when decoding IRAP or GDR images. APS can be used with LMCS, scaling lists, or ALF. Therefore, the decoder determines whether the image is relevant to resynchronization processing, and if so, decoding includes resetting the APS ID.

[0410] In the variant, when one of the stripes in the image is an IRAP, all APS IDs are reset.

[0411] Resetting the APS ID means that the decoder considers all previous APSs to be invalid, so there is no APS ID value to reference previous APSs.

[0412] This resynchronization process is particularly advantageous when combined with the above syntactic structure because it makes it easier to parse and decode the APS ID information, thus enabling faster and more efficient resynchronization when needed.

[0413] Example

[0414] APS ID information in the strip header

[0415] The following description is an example of a modification to the syntactic element table used for the stripe header. In this table, the APSID information of the ALF is set close to the beginning of the stripe header to address the parsing issues discussed above. The modified stripe header in Table 11 is based on the description of the current stripe header of the syntactic elements (see Table 9 above), where modifications are indicated by strikethrough and underscores.

[0416] Table 11 shows the modified partial strip headers.

[0417]

[0418]

[0419]

[0420] APS ID ALF at the beginning of the strip head

[0421] As shown in Table 11, in this embodiment, the APS ID ALF syntactic element set is set at or near the beginning of the strip header. Specifically, the ALF APS contains at least the syntactic elements associated with the signal notification of the clipping value.

[0422] This reduces the complexity for streaming applications that need to track APS ID usage to remove unused APS NAL units.

[0423] A collection of APS ID ALF syntax elements can be set after syntax elements useful for streaming applications that need to parse, cut, split, or extract portions of a video sequence without parsing tool parameters. Examples of such syntax elements are provided below:

[0424] - The slice_pic_order_cnt_lsb syntax element indicates the POC of the current slice. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID ALF syntax elements.

[0425] - The slice_subpic_id syntax element indicates the subpick ID of the current slice. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID ALF syntax elements.

[0426] - The slice_address syntax element indicates the address of the slice. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID ALF syntax elements.

[0427] - num_tiles_in_slice_minus syntax element. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID ALF syntax elements.

[0428] - The slice_type element indicates the type of the current slice. This information is useful for many streaming applications, and therefore it is preferable to set this flag before the set of APS ID ALF syntax elements.

[0429] In an embodiment, when the ALF APS contains at least the syntactic elements associated with the signal notification of the clipping value, the APS ID ALF syntactic element set is set before the low-level tool parameters in the strip header.

[0430] This reduces complexity for some streaming applications that need to track different APS, ALF, and APS IDs for individual images. In fact, the information relevant to low-level tools is needed for parsing and decoding striped data, but is unnecessary for applications that only extract the bitstream without decoding. An example of such a syntax element is provided below:

[0431] - A list of reference images in the strip header. These parameters are actually required for strip data decoding.

[0432] - The set of CABAC initialization flags in the stripe header. In practice, this flag is only used for stripe data decoding.

[0433] - The set of juxtaposed prediction parameters in the strip header. In fact, these parameters are required for strip data decoding.

[0434] - The set of weighted prediction parameters in the strip header. In fact, these parameters are required for strip data decoding.

[0435] - Update the set of quantization parameters in the strip header. In fact, these parameters are required for strip data decoding.

[0436] - The set of SAO enable flags in the stripe header. These parameters are actually required for stripe data decoding.

[0437] It should be understood that the features described above can be provided in combination with each other. Doing so, as with the specific combinations discussed above, can provide specific advantages suitable for a particular implementation; for example, increased flexibility, or specifying a "worst-case" example. In other examples, complexity requirements may have a higher priority than, for example, rate reduction, and therefore features can be implemented individually.

[0438] As shown in Table 11, according to the embodiment, information related to enabling or disabling LMCS at the strip level is set at or near the beginning of the strip header.

[0439] This reduces the complexity for streaming applications that need to track APS ID usage to remove unused APS NAL units. An APS ID associated with an LMCS does not exist within the slice header, but it does affect the APS ID used for the current image when it is disabled (i.e., flagged at the slice level). For example, for sub-image extraction, the APS ID should be sent in the image header, but the extracted sub-image will only contain one slice where the LMCS might be disabled in the slice header. If the APS is not used in another frame, the extraction application should remove the APS LMCS with the associated APS ID because the extracted sub-image does not require that APS.

[0440] Flags related to enabling or disabling LMCS at the strip level are set close to the ALF-related information and preferably after these ALF syntax elements in the syntax table of the strip header. For example, as shown in Table 11, the LMCS flag can immediately follow the ALF syntax element (when enabled). Therefore, the syntax for LMCS can be set (parsed) after syntax elements useful for streaming applications that require parsing, cutting, splitting, or extracting portions of a video sequence without needing to parse all tool parameters. Examples of such syntax elements have been described previously.

[0441] As shown in Table 11, in embodiments of the present invention, information related to enabling or disabling the scaling list at the strip level is set at or near the beginning of the strip header.

[0442] Similar to the approach for LMCS, this reduces the complexity for streaming applications that need to track APS ID usage to remove unused APS NAL units. The APS ID associated with the scaling list is not present in the stripe header, but it has an impact on the APS ID used for the current image when it is disabled (i.e., through information such as flags set at the stripe level). For example, for sub-image extraction, the APS ID should be sent in the image header, but the extracted sub-image will only contain one stripe where the scaling list might be disabled in the stripe header. If that APS is never used in another frame, the extraction application should remove the APS scaling list with the associated APS ID, as the extracted sub-image does not need that APS scaling list.

[0443] Flags related to enabling or disabling scaling lists at the strip level are set close to the ALF-related information, and preferably after these ALF syntax elements in the syntax table of the strip header, and more preferably after the LMCS-related information (e.g., information on enabling or disabling LMCS at the strip level). Therefore, flags related to enabling or disabling scaling lists can be set after syntax elements useful for streaming applications (which need to parse, cut, split, or extract portions of a video sequence without needing to parse all tool parameters). Examples of such syntax elements have been described previously. Therefore, in addition to the APS ID ALF syntax element, the setting / parsing of syntax elements related to LMCS and / or scaling lists can also be performed before the low-level tool parameters mentioned above.

[0444] In this embodiment, the streaming application does not look at the slice-level LMCS flag (slice_lmcs_enabled_flag) to select the correct AP. This reduces complexity compared to previous embodiments because it eliminates the need to parse the slice header for LMCS. However, this potentially increases the bit rate because some APS for the LMCS are sent even if they will not be used for decoding processing. However, this depends on the streaming application. For example, in a bitstream encapsulation packaged as a file format, there is no possibility that the APS LMCS signaled in the image header will never be used in any slice. In other words, in a bitstream encapsulation, we can ensure that the APS LMCS will be used in at least one slice.

[0445] In this embodiment, additionally or alternatively, the streaming application does not look at the slice-level scaling list flag (slice_scaling_list_present_flag) to select the correct APS associated with the scaling list. This reduces complexity compared to previous embodiments because it eliminates the need to parse the slice header for the scaling list. However, sometimes the bit rate is increased because some APS for the scaling list is sent even if it will not be used for decoding processing. However, this depends on the streaming application. For example, in a bitstream encapsulation packaged as a file format, there is no possibility that the APS scaling list signaled in the image header will never be used in any slice. In other words, in a bitstream encapsulation, we can ensure that the APS scaling list will be used in at least one slice.

[0446] In the above embodiments, we refer to LMCS and scaling lists as tools to which the method can be applied. However, the invention is not limited to LMCS and scaling lists. The invention is applicable to any decoding tool or parameter that can be enabled at the image level and obtain APS or other parameters, but subsequently disabled at the stripe level.

[0447] Implementation of the present invention

[0448] Figure 9 Systems 191 and 195 according to embodiments of the present invention are illustrated, comprising at least one of an encoder 150 or a decoder 100 and a communication network 200. According to an embodiment, system 195 is used to process and provide content (e.g., video and audio content for display / output or streaming video / audio content) to a user, who accesses decoder 100, for example, through a user interface of a user terminal including decoder 100 or a user terminal capable of communicating with decoder 100. Such a user terminal may be a computer, mobile phone, tablet computer, or any other type of device capable of providing / displaying (provided / streamed) content to the user. System 195 receives / receives bitstream 101 (in the form of a continuous stream or signal (e.g., when displaying / outputting earlier video / audio)) via communication network 200. According to an embodiment, systems 191 and 195 are used to process and store processed content, such as video and audio content processed for display / output / streaming at a later time. System 191 acquires / receives content including the original image sequence 151, which is received and processed by encoder 150 (including filtering using the deblocking filter according to the invention), and encoder 150 generates a bitstream 101 to be transmitted to decoder 100 via communication network 191. The bitstream 101 is then transmitted to decoder 100 in various ways, for example, it may be pre-generated by encoder 150 and stored as data in a storage device within communication network 200 (e.g., on a server or cloud storage device) until a user requests content (i.e., bitstream data) from the storage device, at which point the data is transferred / streamed from the storage device to decoder 100. Systems 191, 195 may also include content providing devices for providing / streaming (e.g., by transmitting user interface data to be displayed on a user terminal) content information (e.g., content title and other metadata / storage location data for identifying, selecting, and requesting content) of content stored in the storage device to the user, and for receiving and processing user requests for content such that the requested content can be transferred / streamed from the storage device to the user terminal. Alternatively, encoder 150 generates bitstream 101 and transmits / streams it directly to decoder 100 when the user requests content. Decoder 100 then receives bitstream 101 (or signal) and filters it using the deblocking filter according to the invention to obtain / generate video signal 109 and / or audio signal, which the user terminal then uses to provide the requested content to the user.

[0449] Any step of the method / process according to the invention or the function described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the step / function may be stored on or transmitted via one or more hardware-based processing units as one or more instructions, code, programs, or computer-readable media, and executed by one or more hardware-based processing units, such as programmable computing machines, which may be PCs (“personal computers”), DSPs (“digital signal processors”), circuits, circuit systems, processors and memories, general-purpose microprocessors or central processing units, microcontrollers, ASICs (“application-specific integrated circuits”), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, the term “processor” as used herein may refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein.

[0450] Embodiments of the present invention can also be implemented by various devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or JC sets (e.g., chipsets). Various components, modules, or units are described herein to illustrate functional aspects of the apparatus / apparatus configured to perform these embodiments, but they do not necessarily need to be implemented by different hardware units. Rather, various modules / units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors incorporating suitable software / firmware.

[0451] Embodiments of the present invention can be implemented by a computer of a system or device that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium to perform one or more modules / units / functions in the above embodiments and / or includes one or more processing units or circuits for performing one or more functions in the above embodiments. Furthermore, the invention can be implemented by a method performed by the computer of the system or device, such as reading and executing computer-executable instructions from a storage medium to perform one or more functions in the above embodiments and / or controlling one or more processing units or circuits to perform one or more functions in the above embodiments. The computer may include a network of separate computers or separate processing units to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, via a network or tangible storage medium from a computer-readable medium such as a communication medium. The communication medium may be a signal / bit stream / carrier. Tangible storage media are “non-transitory computer-readable storage media”, which may include, for example, one or more of the following: hard disk, random access memory (RAM), read-only memory (ROM), storage device for distributed computing systems, optical disc (e.g., compact disc (CD), digital versatile optical disc (DVD) or Blu-ray disc (BD)™), flash memory device, memory card, etc. At least some steps / functions may also be implemented in hardware by a machine or special-purpose component (such as FPGA (“Field Programmable Gate Array”) or ASIC (“Application-Specific Integrated Circuit”)).

[0452] Figure 10This is a schematic block diagram of a computing device 2000 for implementing one or more embodiments of the present invention. The computing device 2000 may be a device such as a microcomputer, a workstation, or a lightweight portable device. The computing device 2000 includes a communication bus connected to the following: a central processing unit (CPU) 2001, such as a microprocessor; a random access memory (RAM) 2002 for storing executable code of methods according to embodiments of the present invention and registers adapted to record variables and parameters required for implementing methods for encoding or decoding at least a portion of an image according to embodiments of the present invention, the storage capacity of which can be expanded, for example, by connecting optional RAM to an expansion port; a read-only memory (ROM) 2003 for storing a computer program for implementing embodiments of the present invention; a network interface (NET) 2004, which is typically connected to a communication network through which digital data to be processed is transmitted or received, the network interface (NET) 2004 may be a single network interface or a set of different network interfaces (e.g., wired and wireless interfaces, or different kinds of wired or wireless interfaces), through which data packets are written to for transmission or read from for reception under the control of a software application running in the CPU 2001; a user interface (UI) 2005, which can be used to receive input from a user or display information to a user; and a hard disk (HD). 2006, which can be configured as a mass storage device; 2007, an input / output module (IO) for receiving / sending data from / to external devices (such as video sources or displays). Executable code can be stored in ROM 2003, HD 2006, or on a removable digital medium such as a disk. According to a variant, the executable code of the program can be received via NET 2004 through a communication network to be stored in one of the storage components (such as HD 2006) of the communication device 2000 before execution. CPU 2001 is adapted to control and direct the execution of instructions or portions of software code of one or more programs according to embodiments of the present invention, which are stored in one of the aforementioned storage components. For example, upon power-up, CPU 2001 is capable of executing those software application-related instructions from main RAM memory 2002 after instructions have been loaded from program ROM 2003 or HD 2006. Such software application, when executed by CPU 2001, causes the steps of the method according to the present invention to be performed.

[0453] It should also be understood that, according to other embodiments of the invention, a decoder according to the above embodiments is provided in a user terminal such as a computer, mobile phone (cellular phone), tablet, or any other type of device capable of providing / displaying content to a user (e.g., a display device). According to yet another embodiment, an encoder according to the above embodiments is provided in an image capture device, which further includes a camera, video camera, or webcam (e.g., a closed-circuit television or video surveillance camera) for capturing and providing content for encoding by the encoder. See below. Figure 11 and 12 Here are two such examples.

[0454] Webcam

[0455] Figure 11 This is a diagram illustrating a webcam system 2100 including a webcam 2102 and a client device 2104.

[0456] The network camera 2102 includes a camera unit 2106, an encoding unit 2108, a communication unit 2110, and a control unit 2112.

[0457] The network camera 2102 and the client device 2104 are interconnected via network 200 so that they can communicate with each other.

[0458] The camera unit 2106 includes a lens and an image sensor (e.g., a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS)), and captures an image of the object and generates image data based on the image. The image can be a still image or a video image.

[0459] The encoding unit 2108 encodes the image data using the encoding method described above.

[0460] The communication unit 2110 of the network camera 2102 transmits the encoded image data encoded by the encoding unit 2108 to the client device 2104.

[0461] In addition, the communication unit 2110 receives commands from the client device 2104. These commands include those for setting parameters for encoding by the encoding unit 2108.

[0462] The control unit 2112 controls other units in the network camera 2102 according to the commands received by the communication unit 2110.

[0463] The client device 2104 includes a communication unit 2114, a decoding unit 2116, and a control unit 2118.

[0464] The communication unit 2114 of the client device 2104 transmits commands to the network camera 2102.

[0465] In addition, the communication unit 2114 of the client device 2104 receives encoded image data from the webcam 2102.

[0466] Decoding unit 2116 decodes the encoded image data using the decoding method described above.

[0467] The control unit 2118 of the client device 2104 controls other units in the client device 2104 based on user operations or commands received by the communication unit 2114.

[0468] The control unit 2118 of the client device 2104 controls the display device 2120 to display the image decoded by the decoding unit 2116.

[0469] The control unit 2118 of the client device 2104 also controls the display device 2120 to display the values ​​of the parameters for specifying the network camera 2102 (including the parameters for encoding the encoding unit 2108).

[0470] The control unit 2118 of the client device 2104 also controls other units in the client device 2104 based on user operation input to the GUI displayed on the display device 2120.

[0471] The control unit 2118 of the client device 2104 controls the communication unit 2114 of the client device 2104 based on user operation input to the GUI displayed on the display device 2120, so as to transmit commands for specifying the values ​​of parameters of the network camera 2102 to the network camera 2102.

[0472] Smartphone

[0473] Figure 12 This is a diagram illustrating a smartphone 2200.

[0474] The smartphone 2200 includes a communication unit 2202, a decoding unit 2204, a control unit 2206, a display unit 2208, an image recording device 2210, and a sensor 2212.

[0475] The communication unit 2202 receives encoded image data via the network 200.

[0476] Decoding unit 2204 decodes the encoded image data received by communication unit 2202.

[0477] Decoding unit 2204 decodes the encoded image data using the decoding method described above.

[0478] The control unit 2206 controls other units in the smartphone 2200 based on user operations or commands received from the communication unit 2202.

[0479] For example, control unit 2206 controls display unit 2208 to display the image decoded by decoding unit 2204.

[0480] Although the invention has been described with reference to embodiments, it should be understood that the invention is not limited to the disclosed embodiments. Those skilled in the art will understand that various changes and modifications can be made without departing from the scope of the invention as defined by the appended claims. All features disclosed in this specification (including any appended claims, abstract, and drawings), and / or all steps of any disclosed method or process, can be combined in any combination except for at least some mutually exclusive combinations of such features and / or steps. Unless otherwise expressly stated, the various features disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by alternative features for the same, equivalent, or similar purposes. Therefore, unless otherwise expressly stated, the various features disclosed are merely examples of a general series of equivalent or similar features.

[0481] It should also be understood that any result of the above comparisons, determinations, evaluations, selections, executions, processes, or considerations (e.g., selections made during encoding or filtering processes) may be indicated in or determined / inferred from data in the bitstream (e.g., flags or data indicating the result), such that the indicated or determined / inferred result may be used for processing rather than actually being compared, determined, evaluated, selected, executed, processed, or considered, for example, during decoding processes.

[0482] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude multiple elements. The mere fact that different features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used advantageously.

[0483] The reference numerals appearing in the claims are for illustrative purposes only and should not be construed as limiting the scope of the claims.

Claims

1. A method for decoding video data from a bitstream, the bitstream comprising video data corresponding to one or more stripes, wherein, The bitstream includes an image header and a stripe header. The image header includes syntax elements to be used when decoding an image comprising one or more stripes. The stripe header includes syntax elements to be used when decoding a stripe. The method includes: The image header is parsed for at least one syntactic element indicating whether LMCS is enabled for the image, where LMCS is a luminance mapping with chroma scaling. When the LMCS decoding tool is enabled for the image, at least one syntactic element related to the ID of the LMCS APS for the LMCS decoding tool is parsed from the image header, where APS is an adaptive parameter set. When the LMCS decoding tool is enabled for the image, from the stripe header, immediately following the syntax element in the stripe header related to the ID of the ALF APS, and before the syntax element related to one or more decoding tools, at least one syntax element indicating whether to use the LMCS decoding tool for the stripe is parsed, where ALF is an adaptive loop filter, and the syntax element indicating whether to use the LMCS decoding tool for the stripe is a flag signaling whether to use the LMCS decoding tool for the stripe; and The video data is decoded from the bitstream using the syntactic elements.

2. A method for encoding video data into a bitstream, the bitstream comprising video data corresponding to one or more stripes, wherein, The bitstream includes an image header and a stripe header. The image header includes syntax elements to be used when decoding an image comprising one or more stripes. The stripe header includes syntax elements to be used when decoding a stripe. The method includes: In the image header, at least one syntactic element is encoded that indicates whether a decoding tool for LMCS is enabled for the image, where LMCS is a luminance map with chroma scaling. When an LMCS decoding tool is enabled for the image, at least one syntactic element related to the ID of the LMCS APS used by the LMCS decoding tool is encoded in the image header, where the APS is an adaptive parameter set; and When the LMCS decoding tool is enabled for the image, in the strip header, immediately after the syntax element related to the ID of the ALF APS in the strip header, and before the syntax element related to one or more decoding tools, at least one syntax element indicating whether to use the LMCS decoding tool for the strip is encoded, where ALF is an adaptive loop filter, and the syntax element indicating whether to use the LMCS decoding tool for the strip is a flag that signals whether to use the LMCS decoding tool for the strip.

3. An apparatus for decoding video data from a bitstream, the bitstream comprising video data corresponding to one or more stripes, wherein, The bitstream includes an image header and a stripe header. The image header includes syntax elements to be used when decoding an image comprising one or more stripes. The stripe header includes syntax elements to be used when decoding a stripe. The apparatus includes: A component for parsing at least one syntactic element from the image header that indicates whether a decoding tool is enabled for the image, wherein LMCS is a luminance map with chroma scaling; A component for parsing at least one syntactic element from the image header related to the ID of the LMCS APS for the LMCS decoding tool when an LMCS decoding tool is enabled for the image, wherein the APS is an adaptive parameter set. A component for resolving at least one syntax element indicating whether to use an LMCS decoding tool for a stripe when an LMCS decoding tool is enabled for the image, immediately following the syntax element in the stripe header associated with the ID of the ALF APS and preceding the syntax element associated with one or more decoding tools, wherein the ALF is an adaptive loop filter, and wherein the syntax element indicating whether to use an LMCS decoding tool for a stripe is a flag signaling whether to use an LMCS decoding tool for a stripe; and A component for decoding the video data from the bitstream using the syntactic elements.

4. An apparatus for encoding video data into a bit stream, the bit stream comprising video data corresponding to one or more stripes, wherein, The bitstream includes an image header and a stripe header. The image header includes syntax elements to be used when decoding an image comprising one or more stripes. The stripe header includes syntax elements to be used when decoding a stripe. The apparatus includes: A component for encoding at least one syntactic element in the image header that indicates whether a decoding tool for which LMCS is enabled is enabled for the image, wherein LMCS is a luminance map with chroma scaling. A component for encoding at least one syntactic element in the image header related to the ID of the LMCS APS used by the decoding tool for LMCS when LMCS is enabled for the image, wherein the APS is an adaptive parameter set; and A component for encoding at least one syntax element in the strip header, immediately following the syntax element associated with the ID of the ALF APS in the strip header and before the syntax element associated with one or more decoding tools, indicating whether to use the LMCS decoding tool for the strip when the LMCS decoding tool is enabled for the image, wherein the ALF is an adaptive loop filter, and wherein the syntax element indicating whether to use the LMCS decoding tool for the strip is a flag that signals whether to use the LMCS decoding tool for the strip.

5. A computer-readable storage medium carrying one or more executable instructions, which, when executed on a computer or processor, cause the computer or processor to perform the method according to claim 1.

6. A computer-readable storage medium carrying one or more executable instructions, which, when executed on a computer or processor, cause the computer or processor to perform the method according to claim 2.

7. A computer program product comprising a program that causes a computer to perform the method according to claim 1.

8. A computer program product comprising a program that causes a computer to perform the method according to claim 2.

Images