Encoder and corresponding method and device
By analyzing and utilizing the reference layer syntax elements in the video code stream, the format and bit depth consistency prediction between video layers is achieved, the resource waste problem in video data transmission and storage is solved, the encoding and decoding efficiency is improved, and the image is maintained quality.
Patent Information
- Application Number
- CN202210990227.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-01-02
- Filing Date
- 2020-12-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-12-31
AI Technical Summary
Existing video encoding and decoding technologies are difficult to effectively compress video data during transmission and storage, resulting in waste of bandwidth and storage resources. In the case of limited network resources, image quality is difficult to maintain and compression and decompression efficiency is low.
By analyzing the reference layer syntax elements in the encoded video code stream, determine the inter-layer relationship. If the conditions are met, use chroma format or bit depth-related syntax elements to perform inter-layer prediction to ensure the consistency of format and bit depth, thereby achieving more efficient video encoding and decoding.
It improves the compression ratio and decoding efficiency of video data, maintains image quality, simplifies the design process, and is suitable for scenarios with limited bandwidth and high-quality requirements.
Smart Images

Figure CN115567716B_ABST
Abstract
Description
[0001] This application is a divisional application. The application number of the original application is 202080093707.5, and the original application date is December 31, 2020. The entire contents of the original application are incorporated into this application by reference.
[0002] Cross-reference to related applications
[0003] This application claims priority to international application No. PCT / CN2019 / 130804, filed on December 31, 2019, and claims priority to international application No. PCT / CN2020 / 070153, filed on January 2, 2020, the entire contents of which are incorporated by reference into this application. Technical Field
[0004] Embodiments of the present application (the present invention) generally relate to the field of image processing, and more specifically to an encoder, a decoder, and corresponding methods and devices. Background Art
[0005] Video coding (video encoding and decoding) is widely used in digital video applications such as broadcast digital television (TV), video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, digital video discs (DVD) and Blu-ray discs, video content acquisition and editing systems, and camcorders for security applications.
[0006] Even if a video is relatively short, a large amount of video data is required to describe it, which can cause difficulties when the data is to be streamed or otherwise transmitted over a communication network with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted over modern telecommunications networks. Since memory resources may be limited, the size of the video may also be an issue when storing the video in a storage device. Video compression devices typically encode the video data using software and / or hardware on the source side and then transmit or store the video data, thereby reducing the amount of data required to represent the digital video image. A video decompression device that decodes the video data then receives the compressed data on the destination side. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the invention
[0007] Embodiments of the present application provide various devices and methods for encoding and decoding according to the independent claims.
[0008] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the description and the drawings.
[0009] Particular embodiments are outlined in the accompanying independent claims, further embodiments are outlined in the dependent claims.
[0010] According to a first aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream. The method comprises:
[0011] Obtaining a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index k is a direct reference layer of a layer with an index i, and i and k are both integers and greater than or equal to 0;
[0012] Determine, according to the value of the reference layer syntax element, whether a layer with an index j is a reference layer of a layer with an index i, wherein the layer with an index j is a reference layer of the layer with an index k, and j is an integer greater than or equal to 0;
[0013] If a condition is met, the image in the layer with index i is predicted based on the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0014] According to the first aspect, in a possible implementation of the method, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0015] According to a second aspect, the present invention relates to a method for decoding a coded video stream. The method is performed by a device for decoding a coded video stream. The method comprises:
[0016] Obtain a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index of j is a direct reference layer of a layer with an index of i, and i and j are both integers and greater than or equal to 0;
[0017] The image in the layer with index i is predicted based on the layer with index j if a condition is met, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i.
[0018] According to the first aspect or any of the above implementations of the first aspect or the second aspect, in one possible implementation of the method, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS is applied to the layer indexed by j and the layer indexed by i.
[0019] According to the first aspect or any of the above implementations of the first aspect or the second aspect, in one possible implementation of the method, the chroma format-related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0020] According to the first aspect or any of the foregoing implementations of the first aspect or the second aspect or any of the foregoing implementations of the second aspect, in a possible implementation of the method, the method further includes:
[0021] The chroma format related syntax elements applied to the layer with index i and the chroma format related syntax elements applied to the layer with index j are obtained by parsing the encoded video code stream; the condition also includes that the value of the chroma format related syntax elements applied to the layer with index i is the same as the value of the chroma format related syntax elements applied to the layer with index j.
[0022] According to the first aspect or any foregoing implementation of the first aspect, in a possible implementation of the method, the method further includes:
[0023] If the layer with index j is the reference layer of the layer with index i, and the value of the chroma format related syntax element applied to the layer with index i is different from the value of the chroma format related syntax element applied to the layer with index j, stop decoding the encoded video code stream.
[0024] According to the first aspect or any foregoing implementation of the first aspect, in a possible implementation of the method, the method further includes:
[0025] If the layer with index j is not a reference layer of the layer with index i, the layer with index j is not used to predict the image in the layer with index i.
[0026] According to the first aspect or any of the foregoing implementations of the first aspect or the second aspect or any of the foregoing implementations of the second aspect, in a possible implementation of the method, the method further includes:
[0027] If the condition is met, without obtaining the chroma format-related syntax element applied to the layer with index i by parsing the coded video code stream, it is determined that the value of the chroma format-related syntax element applied to the layer with index i is the value of the chroma format-related syntax element applied to the layer with index j.
[0028] According to a third aspect, the present invention relates to a method for decoding a coded video stream. The method is performed by a device for decoding a coded video stream. The method comprises:
[0029] Obtaining a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index k is a direct reference layer of a layer with an index i, and i and k are both integers and greater than or equal to 0;
[0030] According to the value of the reference layer syntax element, determine whether the layer with index j is the reference layer of the layer with index i, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0031] Predicting an image in the layer indexed i based on the layer indexed j if a condition is met, wherein the value of the bit-depth-related syntax element applied to the layer indexed i is the same as the value of the bit-depth-related syntax element applied to the layer indexed j, the condition including that the layer indexed j is a reference layer of the layer indexed i.
[0032] According to the third aspect, in a possible implementation of the method, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0033] According to a fourth aspect, the present invention relates to a method for decoding a coded video stream, the method being performed by a device for decoding a coded video stream. The method comprises:
[0034] Obtain a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index of j is a direct reference layer of a layer with an index of i, and i and j are both integers and greater than or equal to 0;
[0035] Predicting an image in the layer indexed i based on the layer indexed j if a condition is met, wherein the value of the bit-depth-related syntax element applied to the layer indexed i is the same as the value of the bit-depth-related syntax element applied to the layer indexed j, the condition including that the value of the reference layer syntax element indicates that the layer indexed j is a direct reference layer of the layer indexed i.
[0036] According to the third aspect or any of the above implementations of the first aspect or the fourth aspect, in one possible implementation of the method, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS is applied to the layer indexed by j and the layer indexed by i.
[0037] According to the third aspect or any of the above implementations of the first aspect or the fourth aspect, in one possible implementation of the method, the bit depth-related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0038] According to the third aspect or any of the above implementations of the third aspect or the fourth aspect or any of the above implementations of the fourth aspect, in a possible implementation of the method, the method further includes:
[0039] The bit depth-related syntax element applied to the layer with index i and the bit depth-related syntax element applied to the layer with index j are obtained by parsing the encoded video code stream; the condition also includes that a value of the bit depth-related syntax element applied to the layer with index i is the same as a value of the bit depth-related syntax element applied to the layer with index j.
[0040] According to the third aspect or any foregoing implementation of the third aspect, in a possible implementation of the method, the method further includes:
[0041] If the layer with index j is the reference layer of the layer with index i, and the value of the bit depth related syntax element applied to the layer with index i is different from the value of the bit depth related syntax element applied to the layer with index j, stop decoding the encoded video code stream.
[0042] According to the third aspect or any foregoing implementation of the third aspect, in a possible implementation of the method, the method further includes:
[0043] If the layer with index j is not a reference layer of the layer with index i, the layer with index j is not used to predict the image in the layer with index i.
[0044] According to the third aspect or any of the above implementations of the third aspect or the fourth aspect or any of the above implementations of the fourth aspect, in a possible implementation of the method, the method further includes:
[0045] If the condition is met, without obtaining the bit depth-related syntax element applied to the layer with index i by parsing the coded video code stream, determining that the value of the bit depth-related syntax element applied to the layer with index i is the value of the bit depth-related syntax element applied to the layer with index j.
[0046] According to the third aspect or any of the above implementations of the third aspect or the fourth aspect or any of the above implementations of the fourth aspect, in a possible implementation of the method, the bit depth-related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit depth-related syntax element is applied.
[0047] According to a fifth aspect, the present invention relates to a method for encoding a video. The method is performed by a device for encoding a video. The method comprises:
[0048] Determine whether the layer with index j is a direct reference layer of the layer with index i, where i and j are both integers greater than or equal to 0;
[0049] If the layer with index j is the direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i is encoded into a video bitstream, and a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j.
[0050] According to the fifth aspect, in a possible implementation manner of the method, the method further includes:
[0051] If the layer with index j is a direct reference layer of the layer with index i, the image in the layer with index i is predicted based on the layer with index j.
[0052] According to the fifth aspect, in a possible implementation manner of the method, the method further includes:
[0053] If the layer with index j is not a reference layer of the layer with index i, the layer with index j is not used to predict the image in the layer with index i.
[0054] According to a sixth aspect, the present invention relates to a method for encoding a video. The method is performed by a device for encoding a video. The method comprises:
[0055] Determine whether the layer with index j is a direct reference layer of the layer with index i, where i and j are both integers greater than or equal to 0;
[0056] If the layer with index j is the direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i is encoded into the video code stream, and a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j are encoded into the video code stream, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0057] According to the sixth aspect, in a possible implementation manner of the method, the method further includes:
[0058] If the layer with index j is a direct reference layer of the layer with index i, the image in the layer with index i is predicted based on the layer with index j.
[0059] According to the sixth aspect, in a possible implementation manner of the method, the method further includes:
[0060] If the layer with index j is not a reference layer of the layer with index i, the layer with index j is not used to predict the image in the layer with index i.
[0061] According to the sixth aspect or any one of the above implementations of the sixth aspect, in a possible implementation of the method, the bit depth-related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit depth-related syntax element is applied.
[0062] According to a seventh aspect, the present invention relates to a device for decoding a coded video stream. The device comprises:
[0063] an acquiring unit, configured to acquire a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index k is a direct reference layer of a layer with an index i, and i and k are both integers and greater than or equal to 0;
[0064] a determining unit, configured to determine, according to a value of the reference layer syntax element, whether a layer with an index j is a reference layer of the layer with an index i, wherein the layer with an index j is a reference layer of the layer with an index k, and j is an integer greater than or equal to 0;
[0065] A prediction unit, used to: predict the image in the layer with index i according to the layer with index j if a condition is met, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0066] According to the seventh aspect, in a possible implementation of the device, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0067] According to an eighth aspect, the present invention relates to a device for decoding a coded video code stream. The device comprises:
[0068] an acquiring unit, configured to acquire a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index of j is a direct reference layer of a layer with an index of i, and i and j are both integers and are greater than or equal to 0;
[0069] A prediction unit, used to: predict the image in the layer with index i according to the layer with index j if a condition is met, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i.
[0070] According to the seventh aspect or any of the above implementations of the seventh aspect or the eighth aspect, in a possible implementation of the method, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS is applied to the layer indexed by j and the layer indexed by i.
[0071] According to the seventh aspect or any one of the above implementations of the seventh aspect or the eighth aspect, in a possible implementation of the method, the chroma format-related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0072] According to the seventh aspect or any one of the above implementations of the seventh aspect or the eighth aspect or any one of the above implementations of the eighth aspect, in a possible implementation of the method, the acquisition unit is also used to obtain the chroma format-related syntax elements applied to the layer with index i and the chroma format-related syntax elements applied to the layer with index j by parsing the encoded video code stream; the condition also includes that the value of the chroma format-related syntax elements applied to the layer with index i is the same as the value of the chroma format-related syntax elements applied to the layer with index j.
[0073] According to the seventh aspect or any of the above implementations of the seventh aspect, in a possible implementation of the method, the device further includes a stopping unit;
[0074] The stopping unit is used to stop decoding the encoded video code stream if the layer with index j is the reference layer of the layer with index i, and the value of the chroma format related syntax element applied to the layer with index i is different from the value of the chroma format related syntax element applied to the layer with index j.
[0075] According to the seventh aspect or any one of the above implementations of the seventh aspect, in a possible implementation of the method, the acquisition unit is also used for: if the layer with index j is not a reference layer of the layer with index i, not using the layer with index j to predict the image in the layer with index i.
[0076] According to the seventh aspect or any one of the above implementations of the seventh aspect or the eighth aspect or any one of the above implementations of the eighth aspect, in a possible implementation of the method, the determination unit is also used to: if a condition is met, without obtaining the chroma format-related syntax element applied to the layer with index i by parsing the encoded video code stream, determine that the value of the chroma format-related syntax element applied to the layer with index i is the value of the chroma format-related syntax element applied to the layer with index j.
[0077] According to a ninth aspect, the present invention relates to a device for decoding a coded video stream. The device comprises:
[0078] an acquiring unit, configured to acquire a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index k is a direct reference layer of a layer with an index i, and i and k are both integers and greater than or equal to 0;
[0079] a determining unit, configured to determine, according to a value of the reference layer syntax element, whether the layer with index j is a reference layer of the layer with index i, wherein the layer with index j is a reference layer of the layer with index k, and j is an integer greater than or equal to 0;
[0080] A prediction unit, configured to predict an image in the layer indexed i based on the layer indexed j if a condition is met, wherein the value of the bit depth related syntax element applied to the layer indexed i is the same as the value of the bit depth related syntax element applied to the layer indexed j, and the condition includes that the layer indexed j is a reference layer of the layer indexed i.
[0081] According to the ninth aspect, in a possible implementation of the method, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0082] According to a tenth aspect, the present invention relates to a device for decoding a coded video code stream. The device comprises:
[0083] an acquiring unit, configured to acquire a reference layer syntax element by parsing the coded video bitstream, wherein a value of the reference layer syntax element indicates whether a layer with an index of j is a direct reference layer of a layer with an index of i, and i and j are both integers and are greater than or equal to 0;
[0084] A prediction unit, configured to predict an image in the layer indexed i based on the layer indexed j if a condition is met, wherein the value of the bit depth related syntax element applied to the layer indexed i is the same as the value of the bit depth related syntax element applied to the layer indexed j, and the condition includes that the value of the reference layer syntax element indicates that the layer indexed j is a direct reference layer of the layer indexed i.
[0085] According to the ninth aspect or any of the above implementations of the ninth aspect or the tenth aspect, in a possible implementation of the method, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS is applied to the layer with index j and the layer with index i.
[0086] According to the ninth aspect or any of the above implementations of the ninth aspect or the tenth aspect, in a possible implementation of the method, the bit depth-related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0087] According to the ninth aspect or any of the above implementations of the ninth aspect or the tenth aspect or any of the above implementations of the tenth aspect, in a possible implementation of the method, the acquisition unit is further used to obtain the bit depth-related syntax elements applied to the layer with index i and the bit depth-related syntax elements applied to the layer with index j by parsing the encoded video code stream, wherein the condition also includes that the value of the bit depth-related syntax elements applied to the layer with index i is the same as the value of the bit depth-related syntax elements applied to the layer with index j.
[0088] According to the ninth aspect or any of the above implementations of the ninth aspect, in a possible implementation of the method, the device further includes a stopping unit;
[0089] The stopping unit is used to: stop decoding the encoded video code stream if the layer with index j is the reference layer of the layer with index i, and the value of the bit depth related syntax element applied to the layer with index i is different from the value of the bit depth related syntax element applied to the layer with index j.
[0090] According to the ninth aspect or any one of the above implementations of the ninth aspect, in a possible implementation of the method, the prediction unit is also used for: if the layer with index j is not a reference layer of the layer with index i, not using the layer with index j to predict the image in the layer with index i.
[0091] According to the ninth aspect or any of the above implementations of the ninth aspect or the tenth aspect or any of the above implementations of the tenth aspect, in a possible implementation of the method, the determination unit is further used to: without obtaining the bit depth-related syntax element applied to the layer with index i by parsing the encoded video code stream, determine that the value of the bit depth-related syntax element applied to the layer with index i is the value of the bit depth-related syntax element applied to the layer with index j.
[0092] According to the ninth aspect or any of the above implementations of the ninth aspect or the tenth aspect or any of the above implementations of the tenth aspect, in a possible implementation of the method, the bit depth-related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit depth-related syntax element is applied.
[0093] According to an eleventh aspect, the present invention relates to a device for encoding a video. The device comprises:
[0094] a determining unit, configured to determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, wherein i and j are both integers and are greater than or equal to 0;
[0095] A coding unit, used for: if the layer with index j is the direct reference layer of the layer with index i, encoding a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i into a video bitstream, and encoding a chroma format related syntax element applied to the layer with index i and a chroma format related syntax element applied to the layer with index j into the video bitstream, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j.
[0096] According to the eleventh aspect, in a possible implementation of the method, the device further includes a first prediction unit;
[0097] The first prediction unit is configured to: if the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0098] According to the eleventh aspect, in a possible implementation of the method, the device further includes a second prediction unit;
[0099] The first prediction unit is configured to: if the layer with index j is not a reference layer of the layer with index i, not use the layer with index j to predict the image in the layer with index i.
[0100] According to a twelfth aspect, the present invention relates to a device for encoding a video. The device comprises:
[0101] a determining unit, configured to determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, wherein i and j are both integers and are greater than or equal to 0;
[0102] A coding unit, configured to: if the layer with index j is the direct reference layer of the layer with index i, encode into a video bitstream a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i, and encode into the video bitstream a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0103] According to the twelfth aspect, in a possible implementation of the method, the device further includes a first prediction unit;
[0104] The first prediction unit is configured to: if the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0105] According to the twelfth aspect, in a possible implementation of the method, the device further includes a second prediction unit;
[0106] The second prediction unit is configured to: if the layer with index j is not a reference layer of the layer with index i, not use the layer with index j to predict the image in the layer with index i.
[0107] According to the twelfth aspect or any one of the above implementations of the twelfth aspect, in a possible implementation of the method, the bit depth-related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit depth-related syntax element is applied.
[0108] According to a thirteenth aspect, the present application relates to an encoder. The encoder comprises a processing circuit for executing the method according to the fifth aspect or the sixth aspect.
[0109] According to a fourteenth aspect, the present application relates to a decoder. The decoder comprises a processing circuit for executing the method according to the first aspect, the second aspect, the third aspect or the fourth aspect.
[0110] According to a fourteenth aspect, the present application relates to a computer program product comprising program code. When the program code is executed on a computer or a processor, the program code is used to perform the method according to any one of the first to sixth aspects.
[0111] According to a fifteenth aspect, the present application relates to a decoder. The decoder comprises: one or more processors; and a non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program for execution by the one or more processors, wherein when the program is executed by the one or more processors, the program configures the decoder to perform the method according to the first aspect, the second aspect, the third aspect, or the fourth aspect.
[0112] According to a sixteenth aspect, the present application relates to an encoder. The encoder comprises: one or more processors; and a non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program for execution by the one or more processors, wherein when the program is executed by the one or more processors, the program configures the encoder to perform the method according to the fifth aspect or the sixth aspect.
[0113] According to a seventeenth aspect, the present application relates to a non-transitory computer-readable medium comprising program code. When the program code is executed by a computer device, the program code causes the computer device to perform the method according to any one of the first to sixth aspects.
[0114] According to the eighteenth aspect, the present application relates to a non-transitory storage medium. The non-transitory storage medium includes an encoded code stream decoded by an image decoding device, the code stream includes at least one layer of encoded data, the code stream includes at least one layer of encoded data, the code stream also includes a chroma format related syntax element of a layer indexed by i and a chroma format related syntax element of a layer indexed by j, wherein when the layer indexed by j is a reference layer of the layer indexed by i, the value of the chroma format related syntax element of the layer indexed by i is the same as the value of the chroma format related syntax element of the layer indexed by j, and i and j are both integers and greater than or equal to 0.
[0115] According to the nineteenth aspect, the present application relates to a non-transitory storage medium. The non-transitory storage medium includes an encoded code stream decoded by an image decoding device, the code stream includes encoded data of at least one layer, and the code stream also includes a bit depth-related syntax element of a layer indexed by i and a bit depth-related syntax element of a layer indexed by j, wherein when the layer indexed by j is a reference layer of the layer indexed by i, the value of the bit depth-related syntax element of the layer indexed by i is the same as the value of the bit depth-related syntax element of the layer indexed by j, and i and j are both integers and greater than or equal to 0.
[0116] According to the nineteenth aspect, in a possible implementation of the non-transitory storage medium, the bit depth-related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit depth-related syntax element is applied.
[0117] According to any of the above implementations of the eighteenth aspect or the nineteenth aspect or the nineteenth aspect, in a possible implementation of the method, the code stream also includes a reference layer syntax element, and the layer with index j is the reference layer of the layer with index i, including: the value of the reference layer syntax element indicates that the layer with index j is the direct reference layer of the layer with index i.
[0118] According to a twentieth aspect, the present invention relates to a method for decoding a coded video bitstream. The method is performed by a device for decoding a coded video bitstream, and the method comprises: parsing a first syntax element for deriving a maximum allowed number of layers; when a first condition is satisfied, if there is no value of a second syntax element representing a third syntax element in the coded video bitstream, predicting an image in a current layer, wherein the first condition comprises: the value of the first syntax element indicates that the maximum allowed number of layers is 1.
[0119] According to the twentieth aspect, in a possible implementation of the method, the first syntax element is included in the VPS of the encoded video stream and is used to derive the maximum allowed number of layers in each CVS that references the VPS.
[0120] According to the twentieth aspect or any one of the above implementations of the twentieth aspect, in a possible implementation of the method, if the value of the second syntax element representing the third syntax element does not exist in the encoded video code stream, predicting the image in the current layer with the index includes: if the value of the second syntax element representing the third syntax element does not exist in the PH of the SPS referring to the encoded video code stream, predicting the image in the current layer with the index; the second syntax element is included in the SPS.
[0121] According to the twentieth aspect or any one of the above implementations of the twentieth aspect, in a possible implementation of the method, the method further includes: when the first condition is met, determining that the value of the second syntax element representing the third syntax element does not exist in the encoded video code stream.
[0122] According to the twentieth aspect or any one of the above implementations of the twentieth aspect, in a possible implementation of the method, the third syntax element indicates whether a syntax element of the POC MSB value of the current image exists in the PH, and the third syntax element is included in the PH.
[0123] According to the twenty-first aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream, and the method comprises: parsing a first syntax element, wherein the first syntax element indicates whether the layer with index i uses inter-layer prediction, i is an integer and is greater than 0; when a first condition including the value of the first syntax element indicating that the layer with index j is a direct reference layer of the layer with index i is satisfied, if the value of the chroma format-related syntax element of the layer with index i is the same as the value of the chroma format-related syntax element of the layer with index j, predicting the image in the layer with index i, wherein j is an integer and is greater than 0 under the first condition.
[0124] According to the twenty-first aspect, in a possible implementation of the method, the method also includes: when the first condition is met, determining that the value of the chroma format-related syntax element of the layer with index i is the same as the value of the chroma format-related syntax element of the layer with index j.
[0125] According to the twenty-first aspect or any one of the above implementations of the twenty-first aspect, in a possible implementation of the method, the chroma format-related syntax elements include chroma_format_idc or separate_colour_plane_flag.
[0126] According to the twenty-second aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream, and the method comprises: parsing a first syntax element, wherein the first syntax element indicates whether a layer with an index i uses inter-layer prediction, and i is an integer greater than 0; when a first condition including the value of the first syntax element indicating that a layer with an index j is a direct reference layer of the layer with an index i is satisfied, if the value of the bit depth-related syntax element of the layer with an index i is the same as the value of the bit depth-related syntax element of the layer with an index j, predicting an image in the layer with an index i, wherein j is an integer greater than 0 under the first condition.
[0127] According to the twenty-second aspect, in a possible implementation of the method, the method also includes: when the first condition is met, determining that the value of the bit depth-related syntax element of the layer with index i is the same as the value of the bit depth-related syntax element of the layer with index j.
[0128] According to the 22nd aspect or any one of the above implementations of the 22nd aspect, in a possible implementation of the method, the bit depth-related syntax element includes bit_depth_minus8.
[0129] According to a twenty-third aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream, and the method comprises: deriving a maximum allowed number of layers; when a first condition is satisfied, if there is no value of a first syntax element representing a second syntax element in the coded video code stream, predicting an image in a current layer, wherein the first condition comprises that the value of the maximum allowed number of layers is 1.
[0130] According to a twenty-fourth aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream, and the method comprises: obtaining the number of layers in the video code stream; when a first condition including the number of layers being greater than 1 is satisfied, predicting an image in a layer having the same value of a chroma format-related syntax element.
[0131] According to the twenty-fourth aspect, in a possible implementation of the method, the method further includes: when the first condition is met, determining that the values of the chroma format-related syntax elements of the layers are the same.
[0132] According to the 24th aspect or any one of the above implementations of the 24th aspect, in a possible implementation of the method, the chroma format-related syntax elements include chroma_format_idc or separate_colour_plane_flag.
[0133] According to the twenty-fourth aspect or any one of the above implementations of the twenty-fourth aspect, in a possible implementation of the method, obtaining the number of layers in the video bitstream includes: parsing syntax elements used to derive the maximum allowed number of layers to obtain the number of layers (for example, vps_max_layers_minus1).
[0134] According to a twenty-fifth aspect, the present invention relates to a method for decoding a coded video code stream. The method is performed by a device for decoding a coded video code stream, and the method comprises: obtaining the number of layers in the video code stream; when a first condition including the number of layers being greater than 1 is satisfied, predicting images in layers having the same value of bit depth related syntax elements.
[0135] According to the twenty-fifth aspect, in a possible implementation of the method, the method further includes: when the first condition is met, determining that the values of the bit depth-related syntax elements of the layers are the same.
[0136] According to the twenty-fifth aspect or any one of the above implementations of the twenty-fifth aspect, in a possible implementation of the method, the bit depth-related syntax element includes bit_depth_minus8.
[0137] According to the twenty-fifth aspect or any one of the above implementations of the twenty-fifth aspect, in a possible implementation of the method, obtaining the number of layers in the video bitstream includes: parsing the syntax elements used to derive the maximum allowed number of layers to obtain the number of layers (for example, vps_max_layers_minus1).
[0138] According to a twenty-sixth aspect, the present application relates to a decoder, wherein the decoder comprises a processing circuit for executing the method according to any one of the twenty-first to twenty-fifth aspects.
[0139] According to a twenty-seventh aspect, the present application relates to a computer program product comprising a program code. When the program code is executed on a computer or a processor, the program code is used to perform the method according to any one of the twenty-first to twenty-fifth aspects.
[0140] According to a twenty-eighth aspect, the present application relates to a decoder. The decoder comprises: one or more processors; and a non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program for execution by the one or more processors, wherein when the program is executed by the one or more processors, the program configures the decoder to perform the method according to the twenty-first to twenty-fifth aspects.
[0141] According to a twenty-ninth aspect, the present application relates to a non-transitory computer-readable medium comprising a program code. When the program code is executed by a computer device, the program code causes the computer device to perform the method according to any one of the twenty-first to twenty-fifth aspects.
[0142] The following drawings and description set forth one or more embodiments in detail. Other features, objects, and advantages are apparent in the description, drawings, and claims.
[0143] According to an embodiment of the present application, a reference layer syntax element is obtained by parsing the coded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index k is the direct reference layer of the layer with index i, i and k are both integers and greater than or equal to 0; according to the value of the reference layer syntax element, it is determined whether the layer with index j is the reference layer of the layer with index i, wherein the layer with index j is the reference layer of the layer with index k, j is an integer and greater than or equal to 0; if the condition is satisfied, the image in the layer with index i is predicted according to the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is the reference layer of the layer with index i. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design. BRIEF DESCRIPTION OF THE DRAWINGS
[0144] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings and schematic diagrams, in which:
[0145] Figure 1A A block diagram of an example of a video decoding system for implementing an embodiment of the present application;
[0146] Figure 1B A block diagram of another example of a video decoding system for implementing an embodiment of the present application;
[0147] Figure 2 A block diagram of an example of a video encoder for implementing an embodiment of the present application;
[0148] Figure 3 A block diagram of an exemplary structure of a video decoder for implementing an embodiment of the present application;
[0149] Figure 4 A block diagram of an example of an encoding device or a decoding device;
[0150] Figure 5 A block diagram of another example of an encoding device or a decoding device;
[0151] Figure 6 is a schematic diagram of adaptive coding with 2 layers;
[0152] Figure 7 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0153] Figure 8 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0154] Fig. 9 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0155] Fig.10 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0156] Fig.11 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0157] Fig.12 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application;
[0158] Fig.13 A schematic flow chart of a method for encoding a video provided in accordance with an embodiment of the present application;
[0159] Fig.14 A schematic flow chart of a method for encoding a video provided in accordance with an embodiment of the present application;
[0160] Fig.15 A schematic flow chart of a method for encoding a video provided in accordance with an embodiment of the present application;
[0161] Fig.16 A schematic flow chart of a method for encoding a video provided in accordance with an embodiment of the present application;
[0162] Fig.17 A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0163] Fig.18 A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0164] Fig.19 A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0165] Fig. 20 A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0166] Fig.21 A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0167] Fig. 22A structural diagram of a device for decoding a coded video stream provided by one embodiment of the present application;
[0168] Fig.23 A structural diagram of a device for encoding a video provided by an embodiment of the present application;
[0169] Fig.24 A structural diagram of a device for encoding a video provided by an embodiment of the present application;
[0170] Fig.25 A structural diagram of a device for encoding a video provided by an embodiment of the present application;
[0171] Fig.26 A structural diagram of a device for encoding a video provided by an embodiment of the present application;
[0172] Fig. 27 A block diagram of an exemplary structure of a content provision system 3100 for implementing content distribution services;
[0173] Fig.28 A block diagram of an exemplary structure of a terminal device.
[0174] In the following, identical reference numerals refer to identical features or at least functionally equivalent features, unless explicitly stated otherwise. DETAILED DESCRIPTION
[0175] In the following description, reference is made to the accompanying drawings that form a part of the present invention, which illustrate by way of illustration specific aspects of embodiments of the present invention or specific aspects in which embodiments of the present invention may be used. It should be understood that embodiments of the present invention may be used in other aspects and may include structural changes or logical changes not depicted in the accompanying drawings. Therefore, the following detailed description is not to be understood in a limiting sense, and the scope of the present invention is defined by the appended claims.
[0176] For example, it should be understood that the disclosure related to the described method is also applicable to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more steps of multiple steps respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific device is described according to one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more units of multiple units respectively), even if such one or more steps are not explicitly described or shown in the drawings. In addition, it should be understood that, unless otherwise explicitly stated, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.
[0177] Video decoding (coding) generally refers to processing a sequence of images that form a video or video sequence. In the field of video decoding, the terms "frame" and "picture / image" can be used as synonyms. Video decoding (or commonly referred to as decoding) includes two parts: video encoding and video decoding. Video encoding is performed on the source side and generally includes processing (for example, by compression) the original video image to reduce the amount of data required to represent the video image (thereby more efficiently storing and / or transmitting). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "decoding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as involving the "encoding" or "decoding" of the video image or the corresponding video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding).
[0178] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by quantization, etc. to reduce the amount of data representing the video image, and the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.
[0179] Several video coding standards belong to the group of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each image in a video sequence is usually segmented into a set of non-overlapping blocks, and decoding is usually performed at the block level. In other words, the encoder side usually processes the video at the block (video block) level, i.e., encodes it, for example, by using spatial (intra) prediction and / or temporal (inter) prediction to generate a prediction block, subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block, transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed); while the decoder side applies the inverse processing part relative to the encoder to the coded or compressed block to reconstruct the current block for representation. In addition, the encoder replicates the decoder processing cycle, so that the encoder and decoder generate the same prediction (e.g., intra and inter prediction) and / or reconstruction to process, i.e., decode, the subsequent blocks.
[0180] In the following embodiment of the video decoding system 10, the video encoder 20 and the video decoder 30 are based on Figures 1A to 3 Give a description.
[0181] Figure 1A 1 is a schematic block diagram of an exemplary decoding system 10, for example, a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques in the application. The video encoder 20 (or simply an encoder 20) and the video decoder 30 (or simply a decoder 30) in the video decoding system 10 are two examples, that is, devices that can be used to perform various techniques according to various examples described in the application.
[0182] like Figure 1A As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 to a destination device 14 or the like for decoding the encoded image data 21 .
[0183] The source device 12 includes an encoder 20 and may additionally (ie, optionally) include an image source 16 , a pre-processor (or pre-processing unit) 18 (eg, an image pre-processor 18 ), and a communication interface or communication unit 22 .
[0184] The image source 16 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any type of other device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory / storage for storing any of the above images.
[0185] In order to distinguish the pre-processor 18 and the processing performed by the pre-processing unit 18 , the image or image data 17 may also be referred to as a raw image or raw image data 17 .
[0186] The preprocessor 18 is used to receive (raw) image data 17 and perform preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment or denoising, etc. It can be understood that the preprocessing unit 18 may be an optional component.
[0187] The video encoder 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. provide more detailed description).
[0188] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or data obtained after further processing of the encoded image data 21) to another device, such as the destination device 14 or any other device, through the communication channel 13 for storage or direct reconstruction.
[0189] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ), and may additionally (ie, optionally) include a communication interface or communication unit 28 , a post-processor 32 (or a post-processing unit 32 ), and a display device 34 .
[0190] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or data obtained after further processing of the encoded image data 21) directly from the source device 12 or from any other source such as a storage device (such as an encoded image data storage device), and provide the encoded image data 21 to the decoder 30.
[0191] Communication interface 22 and communication interface 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., a wired network or a wireless network or any combination thereof, or any type of private network and public network or any type of combination thereof).
[0192] The communication interface 22 may, for example, be used to package the encoded image data 21 into a suitable format (eg, data packets) and / or process the encoded image data by any type of transmission coding or processing for transmission over a communication link or communication network.
[0193] The communication interface 28 corresponding to the communication interface 22 may, for example, be configured to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing means and / or decapsulation means to obtain the encoded image data 21 .
[0194] Both the communication interface 22 and the communication interface 28 can be configured as Figure 1A The unidirectional communication interface indicated by the arrow of the communication channel 13 pointing from the source device 12 to the destination device 14, or is configured as a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission (such as encoded image data transmission), etc.
[0195] The decoder 30 is used to receive the encoded image data 21 and provide decoded image data 31 or decoded image 31 (hereinafter referred to as Figure 3 or Figure 5 etc. provide more detailed description).
[0196] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.
[0197] The display device 34 in the destination device 14 is used to receive the post-processed image data 33 so as to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for representing the reconstructed image, such as an integrated or external display or display screen. The display can, for example, include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.
[0198] Although Figure 1A The source device 12 and the destination device 14 are shown as separate devices, but the device embodiments may also include both devices or the functions of both devices, that is, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.
[0199] It is obvious to those skilled in the art based on the description that Figure 1A The presence and (precise) functional division of different units or functions within the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.
[0200] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30) or both the encoder 20 and the decoder 30 may be configured to: Figure 1B The processing circuit shown in the figure is implemented. The processing circuit includes one or more microprocessors, one or more digital signal processors (DSP), one or more application-specific integrated circuits (ASIC), one or more field-programmable gate arrays (FPGA), one or more discrete logic, one or more hardware, one or more video decoding dedicated processors or any combination thereof. The encoder 20 can be implemented by the processing circuit 46 to include reference Figure 2 The various modules described in the encoder 20 of FIG. 1 and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented by the processing circuit 46 to include reference Figure 3 The various modules described in the decoder 30 in the embodiment and / or any other decoder system or subsystem described herein. The processing circuit can be used to perform the various operations discussed below. Figure 5 As shown, if the above-mentioned technology is partially implemented in software, a device can store the instructions of the software in a suitable non-transitory computer-readable storage medium, and can execute these instructions in hardware through one or more processors to perform the technology in the present invention. The video encoder 20 or the video decoder 30 can be integrated into a single device as part of a combined encoder / decoder (codec), for example, as Figure 1B shown.
[0201] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or a tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be used for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.
[0202] In some cases, Figure 1A The video decoding system 10 shown is only one example, and the techniques in the present application can be applied to video decoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, data is retrieved from a local memory, streamed over a network, and so on. The video encoding device can encode the data and store the data in a memory, and / or the video decoding device can retrieve the data from the memory and decode the data. In some examples, the encoding and decoding are performed by devices that do not communicate with each other but only encode the data to the memory and / or retrieve the data from the memory and decode the data.
[0203] For ease of description, for example, the present invention is described herein by referring to the reference software of High-Efficiency Video Coding (HEVC) or the next generation video coding standard, Versatile Video Coding (VVC), developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). It is understood by those skilled in the art that the embodiments of the present invention are not limited to HEVC or VVC.
[0204] Encoders and encoding methods
[0205] Figure 2 FIG. 2 is a schematic block diagram of an exemplary video encoder 20 for implementing the techniques in the present application. Figure 2 In the example of , the video encoder 20 includes an input terminal 201 (or an input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or an output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.
[0206] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208 and the mode selection unit 260 may constitute a forward signal path of the encoder 20, and the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 may constitute a backward signal path of the video encoder 20, wherein the backward signal path of the video encoder 20 corresponds to the decoder (see Figure 3The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 also constitute the “built-in decoder” of the video encoder 20.
[0207] Images and image segmentation (images and blocks)
[0208] The encoder 20 may be configured to receive an image 17 (or image data 17) via an input terminal 201 or the like. The image 17 may be an image in a sequence of images constituting a video or a video sequence. The received image or image data may also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be decoded (particularly in video decoding in order to distinguish the current image from other images (e.g., previously encoded and / or decoded images) in the same video sequence (i.e., a video sequence that also includes the current image)).
[0209] A (digital) image is or can be considered as a two-dimensional array or matrix consisting of samples with intensity values. The samples in the array can also be called pixels (pixel / pel) (short for picture elements). The number of samples in the array or image in the horizontal and vertical directions (or axes) defines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or can include three sample arrays. In the RGB format or color space, an image includes corresponding red, green and blue sample arrays. However, in video decoding, each pixel is usually represented in a brightness and chrominance format or color space, such as YCbCr, including a brightness component represented by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luminance, abbreviated as luma) component Y represents the brightness or grayscale intensity (for example, the two are the same in grayscale images), while the two chrominance (chrominance, abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Thus, an image in YCbCr format includes a luminance sample array consisting of luminance sample values (Y) and two chrominance sample arrays consisting of chrominance values (Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa. This process is also called color conversion or conversion. If the image is black and white, the image may include only a luminance pixel array. Accordingly, the image may be, for example, a luminance sample array in black and white format or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.
[0210] An embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 ), for partitioning the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (in H.264 / AVC), or coding tree blocks (CTB) or coding tree units (CTU) (in H.265 / HEVC and VVC). The image partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to vary the block size between images or subsets or groups of images, and to partition each image into a plurality of corresponding blocks.
[0211] In other embodiments, the video encoder may be configured to directly receive the block 203 in the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be decoded.
[0212] Similar to the image 17, the image block 203 is also or can be regarded as a two-dimensional array or matrix composed of samples with intensity values (sample values), but the size of the image block 203 is smaller than that of the image 17. In other words, the block 203 may include, for example, one sample array (e.g., a brightness array in the case of the black and white image 17, or a brightness array or a chrominance array in the case of a color image) or three sample arrays (e.g., one brightness array and two chrominance arrays in the case of the color image 17) or any other number and / or type of arrays, depending on the color format adopted. The number of samples of the block 203 in the horizontal direction and the vertical direction (or axis) defines the size of the block 203. Accordingly, a block may be, for example, an M×N (M columns×N rows) sample array, or an M×N transform coefficient array, etc.
[0213] Figure 2 The illustrated embodiment of the video encoder 20 may be used to encode the image 17 block by block, for example performing encoding and prediction per block 203 .
[0214] Figure 2 The illustrated embodiment of the video encoder 20 may also be used to segment and / or encode an image using slices (also referred to as video slices). An image may be segmented into or encoded using one or more (usually non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles (in H.265 / HEVC and VVC) or bricks (in VVC)).
[0215] Figure 2The illustrated embodiment of the video encoder 20 may also be used to segment and / or encode an image using slice / block groups (also referred to as video block groups) and / or blocks (also referred to as video blocks). An image may be segmented into or encoded using one or more (usually non-overlapping) slice / block groups; each slice / block group may include, for example, one or more blocks (e.g., CTUs) or one or more blocks; each block may, for example, be rectangular and may include one or more complete or partial blocks (e.g., CTUs), etc.
[0216] Residual calculation
[0217] The residual calculation unit 204 can be used to calculate the residual block 205 (also referred to as residual 205) based on the image block 203 and the prediction block 265 (a more detailed description of the prediction block 265 is provided later) in the following manner: for example, the sample value of the prediction block 265 is subtracted from the sample value of the image block 203 sample by sample (pixel by pixel) to obtain the residual block 205 in the sample domain.
[0218] Transform
[0219] The transform processing unit 206 may be used to apply a transform such as discrete cosine transform (DCT) or discrete sine transform (DST) on the sample values of the residual block 205 to obtain a transform coefficient 207 in the transform domain. The transform coefficient 207 may also be referred to as a transform residual coefficient and represents the residual block 205 in the transform domain.
[0220] The transform processing unit 206 can be used to apply an integer approximation of DCT / DST (e.g., a transform specified for H.265 / HEVC). Compared to an orthogonal DCT transform, this integer approximation is typically scaled by a certain factor. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, other scaling factors are applied as part of the transform process. The scaling factor is typically selected based on certain constraints, such as the scaling factor being a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between precision and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform through the inverse transform processing unit 212, etc. (and the video decoder 30 for the corresponding inverse transform through the inverse transform processing unit 312, etc.); accordingly, the encoder 20 can specify a corresponding scaling factor for the forward transform through the transform processing unit 206, etc.
[0221] Embodiments of the video encoder 20 (and accordingly, the transform processing unit 206) may be configured to output transform parameters (e.g., one or more transform types) directly or after encoding or compression through the entropy coding unit 270, such that the video decoder 30 may receive and use the transform parameters for decoding, and so on.
[0222] Quantification
[0223] The quantization unit 208 may be configured to quantize the transform coefficients 207 by applying scalar quantization or vector quantization, etc., to obtain quantized coefficients 209 . The quantized coefficients 209 may also be referred to as quantized transform coefficients 209 or quantized residual coefficients 209 .
[0224] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be performed to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be represented by a quantization parameter (QP). For example, a quantization parameter may be an index of a set of predefined suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, and corresponding inverse quantization and / or inverse dequantization performed by an inverse quantization unit 210 or the like may include multiplying by a quantization step size. Embodiments according to some standards such as HEVC may be used to determine a quantization step size using a quantization parameter. In general, a quantization step size may be calculated according to a quantization parameter by a fixed-point approximation of an equation including division. Additional scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In an exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Optionally, a custom quantization table may be used and signaled by the encoder to the decoder in a bitstream or the like. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.
[0225] Embodiments of the video encoder 20 (correspondingly, the quantization unit 208 ) may be configured to directly output a quantization parameter (QP) or output the quantization parameter after encoding through the entropy coding unit 270 , so that the video decoder 30 may receive and use the quantization parameter for decoding, and so on.
[0226] Dequantization
[0227] The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to the quantized coefficients to obtain dequantized coefficients 211 by, for example, applying an inverse quantization scheme opposite to the quantization scheme applied by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, which correspond to the transform coefficients 207, but are generally different from the transform coefficients due to losses caused by quantization.
[0228] Inverse Transform
[0229] The inverse transform processing unit 212 is used to apply an inverse transform opposite to the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST) or other inverse transforms to obtain a reconstructed residual block 213 (or a corresponding dequantized coefficient 213) in the sample domain. The reconstructed residual block 213 may also be referred to as a transform block 213.
[0230] reconstruction
[0231] The reconstruction unit 214 (e.g., an adder or summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the sample domain by, for example, adding the sample values of the reconstructed residual block 213 and the sample values of the prediction block 265 sample by sample.
[0232] Filtering
[0233] The loop filter unit 220 (or simply referred to as "loop filter" 220) is used to filter the reconstructed block 215 to obtain the filtered block 221, or is generally used to filter the reconstructed sample to obtain the filtered sample value. For example, the loop filter unit is used to smoothly perform pixel conversion or otherwise improve the video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (sample-adaptiveoffset, SAO) filter, or one or more other filters, such as an adaptive loop filter (adaptive loopfilter, ALF), a noise suppression filter (noise suppression filter, NSF) or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before the deblocking filter. In another example, the deblocking filter process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2 is shown as an in-loop filter, but in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstructed block 221.
[0234] An embodiment of the video encoder 20 (correspondingly, the loop filter unit 220) can be used to directly output or output loop filter parameters (such as SAO filter parameters or ALF parameters or LMCS parameters) after encoding by the entropy coding unit 270, so that the decoder 30 can receive and use the same loop filter parameters or the corresponding loop filter for decoding, and so on.
[0235] Decode image buffer
[0236] The decoded picture buffer (DPB) 230 may be a memory for storing reference images or generally storing reference picture data, which are used by the video encoder 20 to encode video data. The DPB 230 may be formed by any of a variety of storage devices, such as a dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM) or other types of storage devices. The decoded picture buffer (DPB) 230 may be used to store one or more filtered blocks 221. The decoded picture buffer 230 may also be used to store other previously filtered blocks (e.g., previously filtered reconstructed blocks 221) in the same current image or a different image (e.g., a previously reconstructed image), and may provide a previously completely reconstructed (i.e., decoded) image (and corresponding reference blocks and samples) and / or a partially reconstructed current image (and corresponding reference blocks and samples) for inter-frame prediction, etc. The decoded picture buffer (DPB) 230 may also be used to store one or more unfiltered reconstructed blocks 215 or generally unfiltered reconstructed samples if the reconstructed blocks 215 are not filtered by the loop filter unit 220, etc., or to store any other blocks or samples obtained after further processing of the reconstructed blocks or samples.
[0237] Mode selection (segmentation and prediction)
[0238] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is used to receive or obtain original image data such as the original block 203 (the current block 203 in the current image 17) and reconstructed image data (such as filtered and / or unfiltered reconstructed samples or blocks in the same (current) image and / or one or more previously decoded images) from the decoded image buffer 230 or other buffers (such as line buffers, not shown in the figure). The reconstructed image data is used as reference image data for prediction such as inter-frame prediction or intra-frame prediction to obtain a prediction block 265 or a prediction value 265.
[0239] The mode selection unit 260 may be used to determine or select a segmentation mode and a prediction mode (e.g., intra-frame or inter-frame prediction mode) for the current block prediction mode (including no segmentation), and generate a corresponding prediction block 265. The prediction block 265 is used to calculate the residual block 205 and reconstruct the reconstructed block 215.
[0240] Embodiments of the mode selection unit 260 may be used to select a partitioning scheme and a prediction mode (e.g., selected from those modes supported or available by the mode selection unit 260). The partitioning scheme and the prediction mode provide the best match or minimum residual (minimum residual means better compression in transmission or storage), or provide minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or consider or balance both of the above. The mode selection unit 260 may be used to determine the partitioning scheme and the prediction mode according to rate distortion optimization (RDO), that is, to select the prediction mode that provides the minimum rate distortion. The terms "best", "minimum", "optimal", etc. in this document do not necessarily mean "best", "minimum", "optimal", etc. in general, but may also refer to situations where termination or selection criteria are met, for example, values exceeding or below a threshold or other constraints may result in a "suboptimal selection", but will reduce complexity and processing time.
[0241] In other words, the partitioning unit 262 can be used to: partition the images in the video sequence into a series of coding tree units (CTUs), for example by iteratively using quad-tree partitioning (QT), binary partitioning (BT) or triple-tree partitioning (TT) or any combination thereof, wherein the CTU 203 can also be partitioned into smaller block partitions or sub-blocks (which again form blocks); and to perform prediction on each block partition or sub-block, etc., wherein the mode selection includes selecting the tree structure of the partition block 203 and selecting the prediction mode applied to each block partition or sub-block.
[0242] The segmentation (eg, performed by segmentation unit 260) and prediction processes (performed by inter-prediction unit 244 and intra-prediction unit 254) performed by exemplary video encoder 20 are described in detail below.
[0243] segmentation
[0244] The partitioning unit 262 can be used to partition an image in a video sequence into a series of coding tree units (CTUs). The partitioning unit 262 can partition (or divide) the coding tree unit (CTU) 203 into smaller partitions, such as smaller blocks of square or rectangular size. For an image with 3 sample arrays, the CTU includes an N×N luma sample block and 2 corresponding chroma sample blocks. The maximum allowed size of a luma block in a CTU is specified as 128×128 in the developing universal video coding (VVC), but may be specified as a value different from 128×128 in the future, such as 256×256. The CTUs in an image can be clustered / grouped into stripe / block groups, blocks, or bricks. Blocks cover rectangular areas of an image, and blocks can be divided into one or more bricks. A brick consists of multiple CTU rows in a block. A block that is not partitioned into multiple bricks can be called a brick. However, a brick is a true subset of a block and is therefore not called a block. VVC supports two block group modes, namely raster scan strip / block group mode and rectangular strip mode. In the raster scan block group mode, a strip / block group includes a series of blocks under the block raster scan in the image. In the rectangular strip mode, a strip includes multiple bricks in the image, which together constitute a rectangular area of the image. The individual bricks in the rectangular strip are arranged in the raster scan order of the bricks in the strip. These small blocks (also called sub-blocks) can be further divided into even smaller partitions. This is also called tree segmentation or hierarchical tree segmentation. The root block at the root tree level 0 (hierarchical level 0, depth 0) can be recursively divided into two or more blocks at the next lower tree level, such as nodes at tree level 1 (hierarchical level 1, depth 1). These blocks can be divided again into two or more blocks at the next lower level, such as tree level 2 (hierarchical level 2, depth 2), etc., until the segmentation ends (because the end criteria are met, such as reaching the maximum tree depth or the minimum block size). Blocks that are not further divided are also called leaf blocks or leaf nodes of the tree. A tree divided into 2 partitions is called a binary tree (binary-tree, BT), a tree divided into 3 partitions is called a ternary tree (ternary-tree, TT), and a tree divided into 4 partitions is called a quadtree (quad-tree, QT).
[0245] For example, a coding tree unit (CTU) may be or may include one CTB consisting of luma samples in an image with three sample arrays and two corresponding CTBs consisting of chroma samples in the image, or may be or may include one CTB consisting of samples in a black and white image or an image decoded using three separate color planes and syntax structures. These syntax structures are used to decode the above samples. Correspondingly, a coding tree block (CTB) may be an N×N sample block, where N may be set to a certain value so that one component is divided into multiple CTBs, which is a partitioning method. A coding unit (CU) may be or may include one coding block consisting of luma samples in an image with three sample arrays and two corresponding coding blocks consisting of chroma samples in the image, or may be or may include one coding block consisting of samples in a black and white image or an image decoded using three separate color planes and syntax structures. These syntax structures are used to decode the above samples. Correspondingly, a coding block (CB) may be an M×N sample block, where M and N may be set to a certain value so that one CTB is divided into multiple coding blocks. This is a segmentation method.
[0246] In an embodiment, for example according to HEVC, a coding tree unit (CTU) may be divided into multiple CUs by a quadtree structure represented as a coding tree. Whether to use inter (temporal) prediction or intra (spatial) prediction to decode an image area is determined at the leaf-CU level. Each leaf-CU may be further divided into 1, 2 or 4 PUs according to the PU partitioning type. The same prediction process is applied within a PU, and relevant information is sent to the decoder in units of PUs. After applying the prediction process according to the PU partitioning type to obtain the residual block, the leaf-CU may be divided into transform units (TUs) according to other quadtree structures similar to the coding tree of the CU.
[0247] In an embodiment, for example, according to the latest video coding standard currently under development called Versatile Video Coding (VVC), a combined quadtree nested multi-type tree (using binary and ternary trees) divides the segmentation structure, for example, for segmenting coding tree units. In the coding tree structure within the coding tree unit, the CU can be square or rectangular. For example, the coding tree unit (CTU) is first segmented by a quadtree. Then, the quadtree leaf nodes can be further segmented by a multi-type tree structure. There are 4 types of segmentation in the multi-type tree structure: vertical binary tree segmentation (SPLIT_BT_VER), horizontal binary tree segmentation (SPLIT_BT_HOR), vertical ternary tree segmentation (SPLIT_TT_VER), and horizontal ternary tree segmentation (SPLIT_TT_HOR). The multi-type leaf nodes are called coding units (CUs), and unless the CU is too large for the maximum transform length, such segmentation is used for prediction and transform processing without any further segmentation. This means that in most cases, CU, PU, and TU have the same block size in the coding block structure of the quadtree nested multi-type tree. This abnormal situation occurs when the maximum supported transform length is less than the width or height of the color component of the CU. VVC has formulated a unique indication mechanism for the split partition information in the coding structure of the quadtree nested multi-type tree. In the indication mechanism, the coding tree unit (CTU) as the root of the quadtree is first split by the quadtree structure. Then, each quadtree leaf node (when large enough to be split) is further split by the multi-type tree structure. In the multi-type tree structure, the first flag (mtt_split_cu_flag) is indicated to indicate whether the node is further split; when the node is further split, the second flag (mtt_split_cu_vertical_flag) is first indicated to indicate the split direction, and then the third flag (mtt_split_cu_binary_flag) is indicated to indicate whether the split is a binary tree split or a ternary tree split. According to the values of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree splitting mode (MttSplitMode) of the CU can be derived by the decoder according to predefined rules or tables. It should be noted that for certain designs, such as the 64×64 luma block and 32×32 chroma pipeline design in the VVC hardware decoder, when the width or height of the luma coding block is greater than 64, TT splitting is prohibited, such as Figure 6As shown. TT splitting is also prohibited when the width or height of the chroma coding block is greater than 32. The pipeline design divides the image into virtual pipeline data units (VPDUs), which are defined as non-overlapping units in the image. In the hardware decoder, consecutive VPDUs are processed simultaneously through multiple pipeline stages. In most pipeline stages, the VPDU size is roughly proportional to the buffer size, so it is important to keep the VPDU size small. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) splitting may increase the VPDU size.
[0248] In addition, it should be noted that when a part of the tree node block exceeds the lower boundary or the right boundary of the image, the tree node block is forcibly divided until all samples of each decoded CU are located within the image boundary.
[0249] For example, the Intra Sub-Partition (ISP) tool can divide the luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions according to the block size.
[0250] In one example, mode selection unit 260 in video encoder 20 may be used to perform any combination of the segmentation techniques described herein.
[0251] As described above, the video encoder 20 is used to determine or select the best or optimal prediction mode from a (eg, predetermined) prediction mode set. The prediction mode set may include intra-frame prediction mode and / or inter-frame prediction mode, etc.
[0252] Intra prediction
[0253] The intra-frame prediction mode set may include 35 different intra-frame prediction modes, for example, non-directional modes or directional modes such as DC (or mean) mode and plane mode defined in HEVC, etc., or may include 67 different intra-frame prediction modes, for example, non-directional modes or directional modes such as DC (or mean) mode and plane mode defined in VVC, etc. For example, several traditional angle intra-frame prediction modes are adaptively replaced with wide-angle intra-frame prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. Moreover, the intra-frame prediction result of the plane mode can also be modified by the position-dependent intra prediction combination (PDPC) method.
[0254] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed samples of neighboring blocks of the same current image according to an intra prediction mode in an intra prediction mode set.
[0255] The intra-frame prediction unit 254 (or generally referred to as the mode selection unit 260) is also used to output the intra-frame prediction parameters (or generally referred to as information representing the selected intra-frame prediction mode of the block) to the entropy coding unit 270 in the form of syntax elements 266 to include the intra-frame prediction parameters in the encoded image data 21, so that the video decoder 30 can receive and use the prediction parameters for decoding, etc.
[0256] Inter prediction
[0257] The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., at least part of the decoded image stored in DPB 230 as described above) and other inter-frame prediction parameters, for example, on whether the entire reference image or only a part of the reference image (e.g. a search window area around the area of the current block) is used to search for the best matching reference block, and / or on whether pixel interpolation is applied, such as half-pixel, quarter-pixel and / or 1 / 16-pixel interpolation.
[0258] In addition to the above prediction modes, skip mode, direct mode and / or other inter-frame prediction modes may also be used.
[0259] For example, using extended fusion prediction, the fusion candidate list in this mode consists of the following five candidate types in order: spatial MVP of spatially adjacent CUs, temporal MVP of collocated CUs, history-based MVP in FIFO table, pairwise average MVP, and zero MV. Moreover, decoder side motion vector refinement (DMVR) based on bilateral matching can be used to improve the accuracy of MV in fusion mode. Merge mode with MVD (MMVD) is derived from merge mode with motion vector difference. The MMVD flag indicates immediately after sending the skip flag and the merge flag to indicate whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be applied. The AMVR scheme supports decoding the MVD of the CU with different precisions. According to the prediction mode of the current CU, the MVD of the current CU can be adaptively selected. When the CU is decoded using the fusion mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The inter-frame prediction signal and the intra-frame prediction signal are weighted averaged to obtain the CIIP prediction signal. Using affine motion compensation prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but the motion vector of the sub-CU within the current CU is predicted. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that requires very little calculation (especially in terms of the number of multiplications and the size of the multipliers). A triangular partitioning mode is used, in which the CU is evenly divided into 2 triangular partitions by diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted averaging of 2 prediction signals.
[0260] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2). The motion estimation unit may be configured to receive or acquire an image block 203 (a current image block 203 in a current image 17) and a decoded image 231, or at least one or more previously reconstructed blocks (e.g., reconstructed blocks in one or more other / different previously decoded images 231) for motion estimation. For example, a video sequence may include a current image and a previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or constitute an image sequence, which image sequence constitutes a video sequence.
[0261] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also referred to as a motion vector (MV).
[0262] The motion compensation unit is used to obtain (e.g., receive) inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 265. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation to achieve sub-pixel accuracy. Interpolation filtering can generate other pixel samples based on known pixel samples, thereby potentially increasing the number of candidate prediction blocks that can be used to decode the image block. When receiving a motion vector corresponding to a PU of a current image block, the motion compensation unit may locate the prediction block pointed to by the motion vector in one of the reference image lists.
[0263] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks in the video slices. In addition to or as an alternative to slices and corresponding syntax elements, partition groups and / or partitions and corresponding syntax elements may be generated or used.
[0264] Entropy Coding
[0265] The entropy coding unit 270 is used to apply or not apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CAVLC), an arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding or other entropy coding methods or techniques) to (uncompressed) quantized coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters and / or other syntax elements to obtain encoded image data 21 that can be output through an output terminal 272 in the form of an encoded bitstream 21, etc., so that the video decoder 30 can receive and use these parameters for decoding, etc. The encoded bitstream 21 can be sent to the video decoder 30, or stored in a memory for later transmission or retrieval by the video decoder 30.
[0266] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal for certain blocks or frames without the transform processing unit 206. In another implementation, the encoder 20 may include a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.
[0267] Decoder and decoding method
[0268] Figure 3 An example of a video decoder 30 for implementing the technology in the present application is shown. The video decoder 30 is used to receive the encoded image data 21 (e.g., the encoded code stream 21) encoded by the encoder 20 (e.g., to obtain a decoded image 331. The encoded image data or code stream includes information for decoding the encoded image data, such as data representing image blocks of encoded video slices (and / or partition groups or partitions) and related syntax elements.
[0269] exist Figure 3In the example of , decoder 30 includes entropy decoding unit 304, inverse quantization unit 310, inverse transform processing unit 312, reconstruction unit 314 (e.g., summer 314), loop filter 320, decoded picture buffer (DPB) 330, mode application unit 360, inter prediction unit 344 and intra prediction unit 354. Inter prediction unit 344 may be or may include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The encoding process described in the video encoder 100 is the inverse of the decoding process.
[0270] As described with reference to the encoder 20, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 344, and the intra prediction unit 354 also constitute the “built-in decoder” of the video encoder 20. Accordingly, the inverse quantization unit 310 may be functionally identical to the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally identical to the inverse transform processing unit 212, the reconstruction unit 314 may be functionally identical to the reconstruction unit 214, the loop filter 320 may be functionally identical to the loop filter 220, and the decoded picture buffer 330 may be functionally identical to the decoded picture buffer 230. Therefore, the explanations made with respect to the corresponding units and functions of the video encoder 20 apply correspondingly to the corresponding units and functions of the video decoder 30.
[0271] Entropy decoding
[0272] The entropy decoding unit 304 is used to parse the code stream 21 (or generally referred to as the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantized coefficients 309 and / or the decoded encoding parameters ( Figure 3 The entropy decoding unit 304 may be used to apply a decoding algorithm or scheme corresponding to the coding scheme described with reference to the entropy coding unit 270 in the encoder 20. The entropy decoding unit 304 may also be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units in the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or as an alternative to the slices and the corresponding syntax elements, the partition groups and / or partitions and the corresponding syntax elements may be received and / or used.
[0273] Dequantization
[0274] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally referred to as information related to inverse quantization) and quantized coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.), and inverse quantize the decoded quantized coefficients 309 according to the quantization parameters to obtain dequantized coefficients 311. The dequantized coefficients 311 may also be referred to as transform coefficients 311. The inverse quantization process may include determining a degree of quantization using a quantization parameter determined by the video encoder 20 for each video block in a video slice (or partition or group of partitions), and also determining a degree of inverse quantization that needs to be applied.
[0275] Inverse Transform
[0276] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311 (also referred to as transform coefficients 311) and transform the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the sample domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.) to determine the transform to be applied to the dequantized coefficients 311.
[0277] reconstruction
[0278] The reconstruction unit 314 (eg, adder or summer 314 ) may be configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the sample domain by, for example, adding sample values of the reconstructed residual block 313 to sample values of the prediction block 365 .
[0279] Filtering
[0280] The loop filter unit 320 (in or after the decoding loop) is used to filter the reconstructed block 315 to obtain the filtered block 321, so as to smoothly perform pixel conversion or otherwise improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before the deblocking filter. In another example, the deblocking filter process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as an in-loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.
[0281] Decode image buffer
[0282] The decoded video blocks 321 in the image are then stored in a decoded image buffer 330, which stores the decoded image 331 as a reference image for subsequent motion compensation and / or output or display of other images.
[0283] The decoder 30 is used to output the decoded image 331 through the output unit 332, etc., so as to be presented to the user or for the user to view.
[0284] predict
[0285] The inter-frame prediction unit 344 may be functionally identical to the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally identical to the intra-frame prediction unit 254, and performs partitioning or segmentation decisions and performs predictions according to the segmentation mode and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304, etc.). The mode application unit 360 may be configured to perform prediction (intra-frame or inter-frame prediction) by block according to the reconstructed image, block or corresponding sample (filtered or unfiltered) to obtain a prediction block 365.
[0286] When the video slice is decoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 is used to generate a prediction block 365 for an image block in the current video slice based on the indicated intra-prediction mode and data from a previously decoded block in the current image. When the video image is decoded as an inter-coded (e.g., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 is used to generate a prediction block 365 for a video block in the current video slice based on a motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, the prediction block may be generated based on one of the reference images in one of the reference image lists. The video decoder 30 may construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. In addition to or as an alternative to slices (e.g., video slices), the same or similar process may be applied to or by embodiments using partition groups (e.g., video partition groups) and / or partitions (e.g., video partitions). For example, video may be coded using I, P, or B partition groups and / or partitions.
[0287] The mode application unit 360 is used to determine prediction information for a video block in a current video slice by parsing motion vectors or related information and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction) for decoding the video blocks in the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information of one or more reference picture lists for the slice, the motion vector for each inter-frame encoded video block of the slice, the inter-frame prediction state for each inter-frame decoded video block of the slice, and other information to decode the video blocks in the current video slice. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar processes can be applied to or by embodiments using block groups (e.g., video block groups) and / or blocks (e.g., video blocks). For example, the video can be decoded using I, P, or B block groups and / or blocks.
[0288] Figure 3 The illustrated embodiment of the video decoder 30 may be used to segment and / or decode an image using slices (also referred to as video slices). An image may be segmented into or decoded using one or more (usually non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs), one or more groups of blocks (e.g., partitions (in H.265 / HEVC and VVC) or bricks (in VVC)).
[0289] Figure 3 The illustrated embodiment of the video decoder 30 may be used to segment and / or decode an image using slice / block groups (also referred to as video block groups) and / or blocks (also referred to as video blocks). An image may be segmented into or decoded using one or more (usually non-overlapping) slice / block groups; each slice / block group may include, for example, one or more blocks (e.g., CTUs) or one or more blocks; each block, for example, may be rectangular and may include one or more complete or partial blocks (e.g., CTUs), etc.
[0290] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal for certain blocks or frames without the inverse transform processing unit 312. In another implementation, the video decoder 30 may include the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.
[0291] It should be understood that the processing result of the current step can be further processed and then output to the next step in the encoder 20 and the decoder 30. For example, after interpolation filtering, motion vector derivation or loop filtering, more operations such as clipping or shifting can be performed on the processing results of interpolation filtering, motion vector derivation or loop filtering.
[0292] It should be noted that more operations can be applied to the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine mode, the planar mode, the ATMVP mode, the temporal motion vector, etc.). For example, the value of the motion vector is constrained within a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is –2^(bitDepth–1)~2^bitDepth–1)–1, where “^” represents a power. For example, if bitDepth is set to 16, the range is –32768~32767; if bitDepth is set to 18, the range is –131072~131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is constrained so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods for constraining motion vectors according to bitDepth are provided here.
[0293] Figure 4 Schematic diagram of a video decoding device 400 provided in accordance with an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder (e.g. Figure 1A 30) or an encoder (e.g. Figure 1A 20 in the video encoder).
[0294] The video decoding device 400 includes an inlet port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data, a processor, a logic unit or a central processing unit (CPU) 430 for processing the data, a transmitting unit (Tx) 440 and an outlet port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the inlet port 410, the receiving unit 420, the transmitting unit 440 and the outlet port 450, which are used as an outlet or an inlet of an optical signal or an electrical signal.
[0295] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more CPU chips, one or more cores (e.g., multi-core processors), one or more FPGAs, one or more ASICs, and one or more DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470. The decoding module 470 implements the disclosed embodiments described above. For example, the decoding module 470 performs, processes, prepares, or provides various decoding operations. Therefore, the inclusion of the decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the conversion of the video decoding device 400 to different states. Optionally, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.
[0296] The memory 460 may include one or more disks, one or more tape drives, and one or more solid-state hard disks, and may be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the programs. For example, the memory 460 may be volatile and / or non-volatile, and may be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).
[0297] Figure 5 A simplified block diagram of an apparatus 500 is provided for one exemplary embodiment. Apparatus 500 may be used as source device 12 and / or destination device 14 in FIG.
[0298] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are currently available or will be developed in the future and are capable of operating or processing information. Although the disclosed implementations may be implemented using a single processor such as the processor 502 shown in the figure, using multiple processors may improve speed and efficiency.
[0299] In one implementation, the memory 504 in the apparatus 500 may be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and an application 510, wherein the application 510 includes at least one program that enables the processor 502 to perform the method described herein. For example, the application 510 may include application 1 to application N, and also include a video decoding application that performs the method described herein.
[0300] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.
[0301] Although the bus 512 in the device 500 is described herein as a single bus, the bus 512 may include multiple buses. In addition, the auxiliary memory 514 may be directly coupled to other components in the device 500 or may be accessed through a network, and may include a single integrated unit (e.g., one memory card) or multiple units (e.g., multiple memory cards). Therefore, the device 500 may be implemented in a variety of configurations.
[0302] Adaptive Coding
[0303] Adaptive coding includes quality adaptability (PSNR adaptability), spatial adaptability, etc. For example, Figure 6 As shown, a sequence can be downsampled to obtain a low spatial resolution version. Both the low spatial resolution version and the original spatial resolution (high spatial resolution) version will be encoded. And in general, the low spatial resolution version will be decoded first and then used as a reference for the subsequently decoded high spatial resolution version.
[0304] In order to describe the information of each layer (quantity, dependencies, output), there is a Video Parameter Set (VPS) defined as follows:
[0305]
[0306]
[0307]
[0308]
[0309] The VPS RBSP shall be available for the decoding process before being referenced, included in at least one AU with TemporalId equal to 0 or provided by an external module.
[0310] All VPS NAL units with a specific vps_video_parameter_set_id value in CVS shall have the same content.
[0311] vps_video_parameter_set_id provides the identifier of the VPS for reference by other syntax elements. The value of vps_video_parameter_set_id should be greater than 0.
[0312] vps_max_layers_minus1+1 represents the maximum allowed number of layers in each CVS of the reference VPS.
[0313] vps_max_sublayers_minus1+1 indicates the maximum number of temporal sublayers that can exist in each CVS of the reference VPS. The value range of vps_max_sublayers_minus1 shall be 0 to 6 (inclusive).
[0314] vps_all_layers_same_num_sublayers_flag equal to 1 indicates that all layers in each CVS of the reference VPS have the same number of temporal sublayers. vps_all_layers_same_num_sublayers_flag equal to 0 indicates that the layers in each CVS of the reference VPS may or may not have the same number of temporal sublayers. When vps_all_layers_same_num_sublayers_flag is not present, the value of vps_all_layers_same_num_sublayers_flag is inferred to be 1.
[0315] vps_all_independent_layers_flag equal to 1 indicates that all layers in the CVS are independently coded without using inter-layer prediction. vps_all_independent_layers_flag equal to 0 indicates that one or more layers in the CVS may use inter-layer prediction. When vps_all_independent_layers_flag is not present, the value of vps_all_independent_layers_flag is inferred to be 1.
[0316] vps_layer_id[i] represents the nuh_layer_id value of layer i. For any two non-negative integer values m and n, when m is less than n, the value of vps_layer_id[m] shall be less than the value of vps_layer_id[n].
[0317] vps_independent_layer_flag[i] equal to 1 indicates that the layer with index i does not use inter-layer prediction. vps_independent_layer_flag[i] equal to 0 indicates that the layer with index i can use inter-layer prediction and the syntax element vps_direct_ref_layer_flag[i][j] (where j ranges from 0 to i–1, inclusive) is present in the VPS. When vps_independent_layer_flag[i] is not present, the value of vps_independent_layer_flag[i] is inferred to be 1.
[0318] vps_direct_ref_layer_flag[i][j] equal to 0 indicates that the layer with index j is not the direct reference layer of the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 indicates that the layer with index j is the direct reference layer of the layer with index i. When vps_direct_ref_layer_flag[i][j] (where i and j range from 0 to vps_max_layers_minus1, inclusive) is not present, vps_direct_ref_layer_flag[i][j] is inferred to be 0. When vps_independent_layer_flag[i] is equal to 0, at least one value of j shall be in the range 0 to i–1, inclusive, such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.
[0319] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] are derived as follows:
[0320]
[0321] The variable GeneralLayerIdx[i] represents the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i], and is derived as follows:
[0322] for(i=0;i<=vps_max_layers_minus1;i++) (38)
[0323] GeneralLayerIdx[vps_layer_id[i]]=i
[0324] each_layer_is_an_ols_flag equal to 1 indicates that each OLS includes only one layer, and each layer in the CVS of the referenced VPS is itself an OLS, where the single layer included in the OLS is the only output layer. each_layer_is_an_ols_flag equal to 0 indicates that the OLS may include more than one layer. If vps_max_layers_minus1 is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be 1. Otherwise, when vps_all_independent_layers_flag is equal to 0, the value of each_layer_is_an_ols_flag is inferred to be 0.
[0325] ols_mode_idc equal to 0 means: the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including the end value), and for each OLS, only the highest layer in the OLS is output.
[0326] ols_mode_idc equal to 1 indicates that the total number of OLSs specified by the VPS is equal to vps_max_layers_minus1+1, the i-th OLS includes layers with layer indices from 0 to i (including the end value), and for each OLS, all layers in the OLS are output.
[0327] ols_mode_idc equal to 2 means that the total number of OLSs specified by the VPS is explicitly indicated, for each OLS the output layer is explicitly indicated, and the other layers are direct or indirect reference layers to the output layer in the OLS.
[0328] The value range of ols_mode_idc shall be 0 to 2 (inclusive). The value 3 of ols_mode_idc is reserved for future use by ITU-T or ISO / IEC.
[0329] When vps_all_independent_layers_flag is equal to 1 and each_layer_is_an_ols_flag is equal to 0, the value of ols_mode_idc is inferred to be 2.
[0330] num_output_layer_sets_minus1+1 indicates the total number of OLSs specified by the VPS when ols_mode_idc is equal to 2.
[0331] The variable TotalNumOlss represents the total number of OLSs specified by VPS and is derived as follows:
[0332]
[0333] ols_output_layer_flag[i][j] equal to 1 indicates that when ols_mode_idc is equal to 2, the layer with nuh_layer_id equal to vps_layer_id[j] is the output layer of the i-th OLS. ols_output_layer_flag[i][j] equal to 0 indicates that when ols_mode_idc is equal to 2, the layer with nuh_layer_id equal to vps_layer_id[j] is not the output layer of the i-th OLS.
[0334] The variable NumOutputLayersInOls[i] represents the number of output layers in the i-th OLS, and the variable OutputLayerIdInOls[i][j] represents the nuh_layer_id value of the j-th output layer in the i-th OLS. These two variables are derived as follows:
[0335]
[0336]
[0337] For each OLS, at least one layer should be an output layer. In other words, for any value of i in the range 0 to TotalNumOlss–1 (inclusive), the value of NumOutputLayersInOls[i] should be greater than or equal to 1.
[0338] The variable NumLayersInOls[i] represents the number of layers in the i-th OLS, and the variable LayerIdInOls[i][j] represents the nuh_layer_id value of the j-th layer in the i-th OLS. These two variables are derived as follows:
[0339]
[0340] Note 1: The 0th OLS includes only the lowest layer (i.e., the layer with nuh_layer_id equal to vps_layer_id[0]), and the only layer included in the 0th OLS is output.
[0341] The variable OlsLayeIdx[i][j] represents the OLS layer index of the layer whose nuh_layer_id is equal to LayerIdInOls[i][j] and is derived as follows:
[0342]
[0343] The lowest layer in each OLS shall be an independent layer. In other words, the value of vps_independent_layer_flag[GeneralLayerIdx[LayerIdInOls[i][0]]] shall be equal to 1 when each i ranges from 0 to TotalNumOlss–1 (inclusive).
[0344] Each layer shall be included in at least one OLS specified by the VPS. In other words, for each layer with a specific nuh_layer_idnuhLayerId value equal to one of vps_layer_id[k] (k ranging from 0 to vps_max_layers_minus1, inclusive), there shall be at least one pair of values of i ranging from 0 to TotalNumOlss–1 (inclusive) and j ranging from NumLayersInOls[i]–1 (inclusive) such that the value of LayerIdInOls[i][j] is equal to nuhLayerId.
[0345] vps_num_ptls indicates the number of syntax structures profile_tier_level() in the VPS.
[0346] pt_present_flag[i] is equal to 1: indicates that the profile, tier and general constraints information is present in the i-th syntax structure profile_tier_level() in the VPS. pt_present_flag[i] is equal to 0: indicates that the profile, tier and general constraints information is not present in the i-th syntax structure profile_tier_level() in the VPS. The value of pt_present_flag[0] is inferred to be 1. When pt_present_flag[i] is equal to 0, the profile, tier and general constraints information in the i-th syntax structure profile_tier_level() in the VPS is inferred to be the same as the profile, tier and general constraints information in the (i–1)-th syntax structure profile_tier_level() in the VPS.
[0347] ptl_max_temporal_id[i] represents the TemporalId of the highest sublayer representation, where the level information of the highest sublayer representation is present in the i-th syntax structure profile_tier_level() in the VPS. The value range of ptl_max_temporal_id[i] shall be 0 to vps_max_sublayers_minus1 (inclusive). When vps_max_sublayers_minus1 is equal to 0, the value of ptl_max_temporal_id[i] is inferred to be 0. When vps_max_sublayers_minus1 is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, the value of ptl_max_temporal_id[i] is inferred to be vps_max_sublayers_minus1.
[0348] vps_ptl_byte_alignment_zero_bit shall be equal to 0.
[0349] ols_ptl_idx[i] represents the index of the syntax structure profile_tier_level() in the list of syntax structures profile_tier_level() in the VPS that applies to the ith OLS when NumLayersInOls[i] is greater than 1. When ols_ptl_idx[i] exists, the value range of ols_ptl_idx[i] shall be 0 to vps_num_ptls–1 (inclusive).
[0350] When NumLayersInOls[i] is equal to 1, the syntax structure profile_tier_level() applicable to the i-th OLS is present in the SPS referenced by the layers in the i-th OLS.
[0351] vps_num_dpb_params indicates the number of syntax structures dpb_parameters() in the VPS. The value range of vps_num_dpb_params shall be 0 to 16 (inclusive). When vps_num_dpb_params does not exist, the value of vps_num_dpb_params is inferred to be 0.
[0352] same_dpb_size_output_or_nonoutput_flag is equal to 1, which means that the syntax element layer_nonoutput_dpb_params_idx[i] does not exist in the VPS. same_dpb_size_output_or_nonoutput_flag is equal to 0, which means that the syntax element layer_nonoutput_dpb_params_idx[i] may or may not exist in the VPS.
[0353] vps_sublayer_dpb_params_present_flag is used to control the presence of the syntax elements max_dec_pic_buffering_minus1[], max_num_reorder_pics[], and max_latency_increase_plus1[] in the syntax structure dpb_parameters() in the VPS. When vps_sub_dpb_params_info_present_flag is not present, vps_sub_dpb_params_info_present_flag is inferred to be 0.
[0354] dpb_size_only_flag[i] equal to 1 indicates that the syntax elements max_num_reorder_pics[] and max_latency_increase_plus1[] do not exist in the i-th syntax structure dpb_parameters() in the VPS. dpb_size_only_flag[i] equal to 0 indicates that the syntax elements max_num_reorder_pics[] and max_latency_increase_plus1[] may exist in the i-th syntax structure dpb_parameters() in the VPS.
[0355] dpb_max_temporal_id[i] represents the TemporalId of the highest sublayer representation, where the DPB parameters of the highest sublayer representation may be present in the i-th syntax structure dpb_parameters() in the VPS. The value of dpb_max_temporal_id[i] shall range from 0 to vps_max_sublayers_minus1, inclusive. When vps_max_sublayers_minus1 is equal to 0, the value of dpb_max_temporal_id[i] is inferred to be 0. When vps_max_sublayers_minus1 is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, the value of dpb_max_temporal_id[i] is inferred to be vps_max_sublayers_minus1.
[0356] layer_output_dpb_params_idx[i] represents the index of the syntax structure dpb_parameters() in the list of syntax structures dpb_parameters() in the VPS that applies to the i-th layer when the i-th layer is the output layer in the OLS. When layer_output_dpb_params_idx[i] exists, the value range of layer_output_dpb_params_idx[i] shall be 0 to vps_num_dpb_params–1 (inclusive).
[0357] If vps_independent_layer_flag[i] is equal to 1, then when the i-th layer is the output layer, the syntax structure dpb_parameters() applicable to the i-th layer is the syntax structure dpb_parameters() present in the SPS referenced by the layer.
[0358] Otherwise (vps_independent_layer_flag[i] is equal to 0), the following applies:
[0359] – When vps_num_dpb_params is equal to 1, the value of layer_output_dpb_params_idx[i] is inferred to be 0.
[0360] – The codestream conformance requirement is that the value of layer_output_dpb_params_idx[i] shall be such that dpb_size_only_flag[layer_output_dpb_params_idx[i]] is equal to 0.
[0361] layer_nonoutput_dpb_params_idx[i] represents the index of the syntax structure dpb_parameters() in the list of syntax structures dpb_parameters() in the VPS that applies to the i-th layer when the i-th layer is a non-output layer in the OLS. When layer_nonoutput_dpb_params_idx[i] exists, the value range of layer_nonoutput_dpb_params_idx[i] shall be 0 to vps_num_dpb_params–1 (inclusive).
[0362] If same_dpb_size_output_or_nonoutput_flag is equal to 1, the following applies:
[0363] – If vps_independent_layer_flag[i] is equal to 1, then when the i-th layer is a non-output layer, the syntax structure dpb_parameters() applicable to the i-th layer is the syntax structure dpb_parameters() present in the SPS referenced by this layer.
[0364] – Otherwise (vps_independent_layer_flag[i] is equal to 0), the value of layer_nonoutput_dpb_params_idx[i] is inferred to be layer_output_dpb_params_idx[i].
[0365] Otherwise (same_dpb_size_output_or_nonoutput_flag is equal to 0), when vps_num_dpb_params is equal to 1, the value of layer_output_dpb_params_idx[i] is inferred to be 0.
[0366] vps_general_hrd_params_present_flag equal to 1 indicates that the syntax structure general_hrd_parameters() and other HRD parameters are present in the VPS RBSP syntax structure. vps_general_hrd_params_present_flag equal to 0 indicates that the syntax structure general_hrd_parameters() and other HRD parameters are not present in the VPS RBSP syntax structure.
[0367] vps_sublayer_cpb_params_present_flag equal to 1 indicates that the i-th syntax structure ols_hrd_parameters() in the VPS includes the HRD parameters of the sublayer representation with TemporalId in the range of 0 to hrd_max_tid[i] (inclusive). vps_sublayer_cpb_params_present_flag equal to 0 indicates that the i-th ols_hrd_parameters() syntax structure in the VPS only includes the HRD parameters of the sublayer representation with TemporalId equal to hrd_max_tid[i]. When vps_max_sublayers_minus1 is equal to 0, the value of vps_sublayer_cpb_params_present_flag is inferred to be 0.
[0368] When vps_sublayer_cpb_params_present_flag is equal to 0, the HRD parameters of the sublayer representation with TemporalId in the range 0 to hrd_max_tid[i]–1 (inclusive) are inferred to be the same as the HRD parameters of the sublayer representation with TemporalId equal to hrd_max_tid[i]. These parameters include the HRD parameters starting from the syntax element fixed_pic_rate_general_flag[i] until the syntax structure sublayer_hrd_parameters(i) immediately following the condition "if (general_vcl_hrd_params_present_flag)" in the syntax structure ols_hrd_parameters.
[0369] num_ols_hrd_params_minus1+1 indicates the number of syntax structures ols_hrd_parameters() present in the syntax structure general_hrd_parameters(). The value of num_ols_hrd_params_minus1 shall be in the range of 0 to 63 (inclusive). When TotalNumOlss is equal to 1, the value of num_ols_hrd_params_minus1 is inferred to be 0.
[0370] hrd_max_tid[i] represents the TemporalId of the highest sublayer representation whose HRD parameters are included in the i-th syntax structure ols_hrd_parameters(). The value of hrd_max_tid[i] shall range from 0 to vps_max_sublayers_minus1, inclusive. When vps_max_sublayers_minus1 is equal to 0, the value of hrd_max_tid[i] is inferred to be 0. When vps_max_sublayers_minus1 is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, the value of hrd_max_tid[i] is inferred to be vps_max_sublayers_minus1.
[0371] ols_hrd_idx[i] represents the index of the syntax structure ols_hrd_parameters() applicable to the i-th OLS. The value of ols_hrd_idx[[i]] shall range from 0 to num_ols_hrd_params_minus1 (inclusive). When ols_hrd_idx[[i]] does not exist, the value of ols_hrd_idx[[i]] is inferred to be 0.
[0372] vps_extension_flag is equal to 0, which means that the syntax element vps_extension_data_flag does not exist in the VPS RBSP syntax structure. vps_extension_flag is equal to 1, which means that the syntax element vps_extension_data_flag exists in the VPS RBSP syntax structure.
[0373] vps_extension_data_flag may take any value. Its presence and value do not affect decoder conformance to the profile specified in this version of this specification. Decoders conforming to this version of this specification shall ignore all syntax elements vps_extension_data_flag.
[0374] DPB management and reference image identification
[0375] In order to manage the reference images in the decoding process, the decoded images need to be saved in the decoded picture buffer (DPB) for reference in subsequent image decoding. To indicate these images, their picture order count (POC) information needs to be indicated directly or directly in the slice header. Usually there are two reference image lists, namely list 0 and list 1. And it is also necessary to include a reference image index to indicate the image in the list. For unidirectional prediction, the reference image is obtained from one reference image list, and for bidirectional prediction, the reference image is obtained from two reference image lists.
[0376] All reference pictures are stored in the DPB. All pictures in the DPB are identified as "used for long-term reference", "used for short-term reference" or "not used for reference", i.e. one of three states. Once a picture is identified as "not used for reference", it is no longer used for reference. If it does not need to be stored for output, it can be removed from the DPB. The status of the reference picture can be indicated in the slice header or derived from the slice header information.
[0377] A new reference picture management method is proposed, called the reference picture list (RPL) method. RPL proposes one or more complete reference picture sets corresponding to the current coding picture. The reference pictures in the reference picture set are used for decoding the current picture or future (subsequent) pictures. Therefore, RPL reflects the picture information in the DPB. Even if the reference picture is not used for reference of the current picture, it still needs to be stored in the RPL if it is used for reference of subsequent pictures.
[0378] After the image is reconstructed, it is stored in the DPB and marked as "for short-term reference" by default. The DPB management operation will start after parsing the RPL information in the slice header.
[0379] Reference image list construction
[0380] Reference picture information may be indicated by a slice header. In addition, there may be some RPL candidates in a sequence parameter set (SPS). In this case, the slice header may include a RPL index to obtain the required RPL information without indicating the entire RPL syntax structure. Alternatively, the entire RPL syntax structure may be indicated in the slice header.
[0381] The RPL method is described as follows.
[0382] In order to reduce the cost bit of RPL indication, there can be some RPL candidates in SPS. The picture can use the RPL index (ref_pic_list_idx[i]) to obtain its RPL information from SPS. The RPL candidate indication is as follows:
[0383]
[0384] The semantics are as follows:
[0385] rpl1_same_as_rpl0_flag equal to 1 means that the syntax structures num_ref_pic_lists_in_sps[1] and ref_pic_list_struct(1,rplsIdx) do not exist and the following apply:
[0386] – The value of num_ref_pic_lists_in_sps[1] is inferred to be the value of num_ref_pic_lists_in_sps[0].
[0387] – The value of each syntax element in ref_pic_list_struct(1, rplsIdx) is inferred to be the value of the corresponding syntax element in ref_pic_list_struct(0, rplsIdx) (where rplsIdx ranges from 0 to num_ref_pic_lists_in_sps[0]–1).
[0388] num_ref_pic_lists_in_sps[i] represents the number of syntax structures ref_pic_list_struct(listIdx, rplsIdx) with listIdx equal to i included in the SPS. The value range of num_ref_pic_lists_in_sps[i] is 0 to 64 (inclusive).
[0389] In addition to obtaining RPL information from the SPS according to the RPL index, the RPL information can also be indicated in the stripe header.
[0390]
[0391]
[0392] ref_pic_list_sps_flag[i] equal to 1 means that the reference picture list i of the current slice is derived from one of the syntax structures ref_pic_list_struct(listIdx, rplsIdx) with listIdx equal to i in the SPS. ref_pic_list_sps_flag[i] equal to 0 means that the reference picture list i of the current slice is derived from the syntax structure ref_pic_list_struct(listIdx, rplsIdx) with listIdx equal to i directly included in the slice header of the current picture.
[0393] When ref_pic_list_sps_flag[i] is not present, the following applies:
[0394] – If num_ref_pic_lists_in_sps[i] is equal to 0, the value of ref_pic_list_sps_flag[i] is inferred to be 0.
[0395] – Otherwise (num_ref_pic_lists_in_sps[i] is greater than 0), if rpl1_idx_present_flag is equal to 0, the value of ref_pic_list_sps_flag[1] is inferred to be ref_pic_list_sps_flag[0].
[0396] – Otherwise, the value of ref_pic_list_sps_flag[i] is inferred to be pps_ref_pic_list_sps_idc[i]–1.
[0397] ref_pic_list_idx[i] represents the index in the list of syntax structures ref_pic_list_struct(listIdx,rplsIdx) with listIdx equal to i included in the SPS for deriving the reference picture list i of the current picture. The syntax element ref_pic_list_idx[i] is represented by Ceil(Log2(num_ref_pic_lists_in_sps[i])) bits. When ref_pic_list_idx[i] does not exist, the value of ref_pic_list_idx[i] is inferred to be 0. The value range of ref_pic_list_idx[i] should be 0 to num_ref_pic_lists_in_sps[i]–1 (inclusive). When ref_pic_list_sps_flag[i] is equal to 1 and num_ref_pic_lists_in_sps[i] is equal to 1, the value of ref_pic_list_idx[i] is inferred to be 0. When ref_pic_list_sps_flag[i] is equal to 1 and rpl1_idx_present_flag is equal to 0, the value of ref_pic_list_idx[1] is inferred to be ref_pic_list_idx[0].
[0398] The variable RplsIdx[i] is derived as follows:
[0399] RplsIdx[i]=ref_pic_list_sps_flag[i]? ref_pic_list_idx[i]:
[0400] num_ref_pic_lists_in_sps[i] (7-95)
[0401] slice_poc_lsb_lt[i][j] represents the value of the picture order number of the j-th LTRP entry in the i-th reference picture list modulo MaxPicOrderCntLsb. The length of the syntax element slice_poc_lsb_lt[i][j] is (log2_max_pic_order_cnt_lsb_minus4+4) bits.
[0402] The variable PocLsbLt[i][j] is derived as follows:
[0403] PocLsbLt[i][j]=ltrp_in_slice_header_flag[i][RplsIdx[i]]? (7-96)
[0404] slice_poc_lsb_lt[i][j]:rpls_poc_lsb_lt[listIdx][RplsIdx[i]][j]
[0405] delta_poc_msb_present_flag[i][j] is equal to 1, indicating that delta_poc_msb_cycle_lt[i][j] exists. delta_poc_msb_present_flag[i][j] is equal to 0, indicating that delta_poc_msb_cycle_lt[i][j] does not exist.
[0406] Assume that prevTid0Pic is the previous picture in decoding order, where the previous picture has the same nuh_layer_id as the current picture, the previous picture's TemporalId is equal to 0, and the previous picture is not a RASL picture or a RADL picture. Assume that setOfPrevPocVals is the set consisting of:
[0407] –PicOrderCntVal of prevTid0Pic,
[0408] – PicOrderCntVal of each picture referenced by an entry in RefPicList[0] or RefPicList[1] of prevTid0Pic and having the same nuh_layer_id as the current picture,
[0409] – PicOrderCntVal of each picture after prevTid0Pic in decoding order, with the same nuh_layer_id as the current picture and before the current picture in decoding order.
[0410] When there is more than one value in setOfPrevPocVals, where the value is equal to PocLsbLt[i][j] modulo MaxPicOrderCntLsb, the value of delta_poc_msb_present_flag[i][j] shall be equal to 1.
[0411] delta_poc_msb_cycle_lt[i][j] represents the value of the variable FullPocLt[i][j], as shown below:
[0412]
[0413] FullPocLt[i][j]=PicOrderCntVal–DeltaPocMsbCycleLt[i][j]*MaxPicOrderCntLsb–(PicOrderCntVal&(MaxPicOrderCntLsb–1))+PocLsbLt[i][j]
[0414] delta_poc_msb_cycle_lt[i][j] should be in the range of 0 to 2 (32 –log2_max_pic_order_cnt_lsb_minus4–4) (Inclusive of the end value.) When delta_poc_msb_cycle_lt[i][j] is not present, the value of delta_poc_msb_cycle_lt[i][j] is inferred to be 0.
[0415] The grammatical structure of RPL is as follows:
[0416]
[0417]
[0418] num_ref_entries[listIdx][rplsIdx] indicates the number of entries in the syntax structure ref_pic_list_struct(listIdx, rplsIdx). The value range of num_ref_entries[listIdx][rplsIdx] should be 0 to sps_max_dec_pic_buffering_minus1+14 (including the end value).
[0419] ltrp_in_slice_header_flag[listIdx][rplsIdx equal to 0 indicates that the POC LSB of the LTRP entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) exists in the syntax structure ref_pic_list_struct(listIdx,rplsIdx). ltrp_in_slice_header_flag[listIdx][rplsIdx] equal to 1 indicates that the POC LSB of the LTRP entry in the syntax structure ref_pic_list_struct(listIdx,rplsIdx) does not exist in the syntax structure ref_pic_list_struct(listIdx,rplsIdx).
[0420] inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 1, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is an ILRP entry. inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 0, indicating that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is not an ILRP entry. When inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is not present, the value of inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is inferred to be 0.
[0421] st_ref_pic_flag[listIdx][rplsIdx][i] equal to 1 indicates that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is a STRP entry. st_ref_pic_flag[listIdx][rplsIdx][i] equal to 0 indicates that the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is a LTRP entry. When inter_layer_ref_pic_flag[listIdx][rplsIdx][i] is equal to 0 and st_ref_pic_flag[listIdx][rplsIdx][i] is not present, the value of st_ref_pic_flag[listIdx][rplsIdx][i] is inferred to be 1.
[0422] The variable NumLtrpEntries[listIdx][rplsIdx] is derived as follows:
[0423]
[0424] abs_delta_poc_st[listIdx][rplsIdx][i] represents the value of the variable AbsDeltaPocSt[listIdx][rplsIdx][i], as shown below:
[0425]
[0426]
[0427] AbsDeltaPocSt[listIdx][rplsIdx][i]=abs_delta_poc_st[listIdx][rplsIdx][i]+1abs_delta_poc_st[listIdx][rplsIdx][i] should be in the range of 0 to 2. 15 –1 (inclusive).
[0428] strp_entry_sign_flag[listIdx][rplsIdx][i] equal to 1 indicates that the value of the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is greater than or equal to 0. strp_entry_sign_flag[listIdx][rplsIdx][i] equal to 0 indicates that the value of the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) is less than 0. When strp_entry_sign_flag[listIdx][rplsIdx][i] does not exist, the value of strp_entry_sign_flag[listIdx][rplsIdx][i] is inferred to be 1.
[0429] The list DeltaPocValSt[listIdx][rplsIdx] is derived as follows:
[0430]
[0431] rpls_poc_lsb_lt[listIdx][rplsIdx][i] represents the value of the picture order number of the picture referenced by the i-th entry in the syntax structure ref_pic_list_struct(listIdx, rplsIdx) modulo MaxPicOrderCntLsb. The length of the syntax element rpls_poc_lsb_lt[listIdx][rplsIdx][i] is (log2_max_pic_order_cnt_lsb_minus4+4) bits.
[0432] Some brief description of the RPL structure
[0433] There is one RPL structure for each list. First, num_ref_entries[listIdx][rplsIdx] is indicated to indicate the number of reference images in the list. ltrp_in_slice_header_flag[listIdx][rplsIdx] is used to indicate whether the least significant byte (LSB) information is indicated in the slice header. If the current reference image is not an inter-layer reference image, st_ref_pic_flag[listIdx][rplsIdx][i] is used to indicate whether the current reference image is a long-term reference image. If it is a short-term reference image, POC information (abs_delta_poc_st and strp_entry_sign_flag) is indicated. If ltrp_in_slice_header_flag[listIdx][rplsIdx] is 0, rpls_poc_lsb_lt[listIdx][rplsIdx][j++] is used to derive the LSB information of the current reference image. The most significant bit (MSB) can be derived directly or inferred based on the information in the slice header (delta_poc_msb_present_flag[i][j] and delta_poc_msb_cycle_lt[i][j]).
[0434] Decoding process for reference picture list construction
[0435] This process is called at the beginning of the decoding process for each slice in a non-IDR picture.
[0436] Reference pictures are addressed by reference indices. A reference index is an index into a reference picture list. When decoding an I slice, the decoding of the slice data does not use the reference picture list. When decoding a P slice, the decoding of the slice data uses only reference picture list 0 (i.e., RefPicList[0]). When decoding a B slice, the decoding of the slice data uses both reference picture list 0 and reference picture list 1 (i.e., RefPicList[1]).
[0437] At the beginning of the decoding process of each slice in a non-IDR picture, reference picture lists RefPicList[0] and RefPicList[1] are derived. The reference picture lists are used in reference picture identification as detailed in clause 8.3.3 or in the decoding of slice data.
[0438] NOTE 1: For an I slice that is not the first slice in a non-IDR picture, RefPicList[0] and RefPicList[1] can be derived for codestream consistency checking purposes, but their derivation is not necessarily applicable to the decoding of the current picture or pictures that follow the current picture in decoding order. For a P slice that is not the first slice in a picture, RefPicList[1] can be derived for codestream consistency checking purposes, but its derivation is not necessarily applicable to the decoding of the current picture or pictures that follow the current picture in decoding order.
[0439] The reference picture lists RefPicList[0] and RefPicList[1], the reference picture scaling ratios RefPicScale[i][j][0] and RefPicScale[i][j][1], and the reference picture scaling flags RefPicIsScaled[0] and RefPicIsScaled[1] are constructed as follows:
[0440]
[0441]
[0442] In the case where each i is equal to 0 or 1, the first NumRefIdxActive[i] entries in RefPicList[i] are called active entries in RefPicList[i], and the other entries in RefPicList[i] are called inactive entries in RefPicList[i].
[0443] NOTE 2: A picture may be referenced by both an entry in RefPicList[0] and an entry in RefPicList[1]. A picture may also be referenced by more than one entry in RefPicList[0] or more than one entry in RefPicList[1].
[0444] NOTE 3: The active entries in RefPicList[0] and the active entries in RefPicList[1] jointly reference all reference pictures that can be used for inter-frame prediction of the current picture and one or more pictures that follow the current picture in decoding order. The inactive entries in RefPicList[0] and the inactive entries in RefPicList[1] jointly reference all reference pictures that are not used for inter-frame prediction of the current picture but can be used for inter-frame prediction of one or more pictures that follow the current picture in decoding order.
[0445] NOTE 4: There can be one or more entries in RefPicList[0] or RefPicList[1] that are equal to "no reference picture" because there is no corresponding picture in the DPB. Each inactive entry in RefPicList[0] or RefPicList[0] that is equal to "no reference picture" should be ignored. Unexpected picture loss should be inferred for each active entry in RefPicList[0] or RefPicList[1] that is equal to "no reference picture".
[0446] The requirement for codestream consistency is that the following constraints apply:
[0447] – For each case where i is equal to 0 or 1, num_ref_entries[i][RplsIdx[i]] shall not be less than NumRefIdxActive[i].
[0448] – A picture referenced by each active entry in RefPicList[0] or RefPicList[1] shall exist in the DPB and have a TemporalId less than or equal to the TemporalId of the current picture.
[0449] – Each picture referenced by an entry in RefPicList[0] or RefPicList[1] shall not be the current picture and the non_reference_picture_flag of that picture shall be equal to 0.
[0450] – A STRP entry in RefPicList[0] or RefPicList[1] for a slice in a picture and an LTRP entry in RefPicList[0] or RefPicList[1] for the same slice or a different slice in the same picture shall not refer to the same picture.
[0451] – There is no LTRP entry in RefPicList[0] or RefPicList[1] where the difference between the PicOrderCntVal of the current picture and the PicOrderCntVal of the picture referenced by this entry is greater than or equal to 2 24 .
[0452] – Assume that setOfRefPics is the unique set of pictures referenced by all entries in RefPicList[0] with the same nuh_layer_id as the current picture and all entries in RefPicList[1] with the same nuh_layer_id as the current picture. The number of pictures in setOfRefPics shall be less than or equal to MaxDecPicBuffMinus1, and setOfRefPics shall be the same for all slices in a picture.
[0453] – When the current picture is a STSA picture, there shall be no active entry in RefPicList[0] or RefPicList[1] with the same TemporalId as the current picture.
[0454] – When the current picture is a picture that follows a STSA picture with the same TemporalId as the current picture in decoding order, there shall be no picture with the same TemporalId as the current picture that is an active entry in RefPicList[0] or RefPicList[1] that precedes the STSA picture in decoding order.
[0455] – When the current picture is a CRA picture, there shall be no pictures referenced by entries in RefPicList[0] or RefPicList[1] that precede any previous IRAP picture in decoding order (if any) in output order or decoding order.
[0456] – When the current picture is a subsequent picture, there shall be no pictures referenced by active entries in RefPicList[0] or RefPicList[1] that were generated by a decoding process for reference pictures that were not available for generating the IRAP picture associated with the current picture.
[0457] – When the current picture is a subsequent picture that follows one or more preceding pictures (if any) associated with the same IRAP picture in both decoding order and output order, there shall be no pictures referenced by entries in RefPicList[0] or RefPicList[1] that were generated by the decoding process for which a reference picture was not available for generating the IRAP picture associated with the current picture.
[0458] – When the current image is a recovery point image or an image whose output order follows the recovery point image, there should be no entry in RefPicList[0] or RefPicList[1] that includes an image generated using a reference image for which the GDR image used to generate the recovery point image is unavailable.
[0459] – When the current picture is a subsequent picture, there shall be no pictures referenced by active entries in RefPicList[0] or RefPicList[1] that precede the associated IRAP picture in output order or decoding order.
[0460] – When the current picture is a subsequent picture that follows one or more preceding pictures (if any) associated with the same IRAP picture in both decoding order and output order, there shall be no pictures referenced by entries in RefPicList[0] or RefPicList[1] that precede the associated IRAP picture in output order or decoding order.
[0461] – When the current picture is a RADL picture, there shall be no entry in RefPicList[0] or RefPicList[1] that is any of the following:
[0462] οRASL images
[0463] o Pictures generated by a decoding process used to generate unavailable reference pictures
[0464] o Pictures that precede the associated IRAP picture in decoding order
[0465] – The picture referenced by each ILRP entry in RefPicList[0] or RefPicList[1] of a slice in the current picture shall be in the same AU as the current picture.
[0466] – A picture referenced by each ILRP entry in RefPicList[0] or RefPicList[1] of a slice in the current picture shall be present in the DPB and the nuh_layer_id of that picture shall be less than the nuh_layer_id of the current picture.
[0467] – Each ILRP entry in the stripe’s RefPicList[0] or RefPicList[1] shall be an active entry.
[0468] After the RPL is constructed, the identification process is as follows.
[0469] Decoding process for reference image identification
[0470] This process is called once per picture after the slice header decoding and the reference picture list construction decoding process for the slice (as detailed in Section 8.3.2), but before the slice data decoding. This process can cause one or more reference pictures in the DPB to be marked as "not used for reference" or "used for long-term reference".
[0471] A decoded picture in the DPB may be identified as "not used for reference", "used for short-term reference", or "used for long-term reference", but may only be identified in one of these three states at any one time within the operation of the decoding process. Assigning one of these identification states to a picture implicitly removes the other identification state (if applicable). When a picture is identified as "used for reference", this uniformly refers to identifying the picture as either "used for short-term reference" or "used for long-term reference" (but not both at the same time).
[0472] STRP and ILRP are determined by their nuh_layer_id and PicOrderCntVal values. LTRP are determined by their nuh_layer_id value and Log2(MaxLtPicOrderCntLsb) LSBs of their PicOrderCntVal value.
[0473] If the current picture is a CLVSS picture, all reference pictures currently included in the DPB that have the same nuh_layer_id as the current picture are marked as "unused for reference".
[0474] Otherwise, the following applies:
[0475] – For each LTRP entry in RefPicList[0] or RefPicList[1], when the referenced picture is a STRP with the same nuh_layer_id as the current picture, the picture is identified as “used for long-term reference”.
[0476] - Each reference picture in the DPB with the same nuh_layer_id as the current picture that is not referenced by any entry in RefPicList[0] or RefPicList[1] is marked as “unused for reference”.
[0477] – For each ILRP entry in RefPicList[0] or RefPicList[1], the referenced picture is identified as “used for long-term reference”.
[0478] Unsynchronized IRAP pictures can cross layers. To support this design, the following POC design is used to mix IRAP pictures and non-IRAP pictures within an AU.
[0479] For independent layers, the POC MSB cycle period indication is used. There is a flag in the SPS that controls whether the picture header has ph_poc_msb_cycle_present_flag, and there is also a length in the SPS. When ph_poc_msb_cycle_present_flag is equal to 1, the POC MSB cycle period after u(v) decoding is indicated in the picture header. When the POC MSB cycle period exists, the POC MSB of the picture is set to poc_msb_cycle*MaxPicOrderCntLsb.
[0480] For dependent layers, if there is a picture picA in the same AU in the reference layer of the current layer, the POC is derived as the POC of picA, and the POC LSB values are required to be aligned across layers. Otherwise, the current POC derivation process applies.
[0481]
[0482]
[0483] bit_depth_minus8 represents the bit depth BitDepth of samples in the luma and chroma arrays and the value of the luma and chroma quantization parameter range offset QpBdOffset as follows:
[0484] BitDepth=8+bit_depth_minus8 (45)
[0485] QpBdOffset=6*bit_depth_minus8 (46)
[0486] bit_depth_minus8 should range from 0 to 8 (inclusive).
[0487] sps_decoding_parameter_set_id indicates the dps_decoding_parameter_set_id value of the DPS referenced by the SPS when it is greater than 0. When sps_decoding_parameter_set_id is equal to 0, the SPS does not reference the DPS and the DPS is not referenced when decoding each CLVS that references the SPS. All SPSs referenced by coded pictures in a codestream should have the same sps_decoding_parameter_set_id value.
[0488] sps_video_parameter_set_id, when greater than 0, indicates the vps_video_parameter_set_id value of the VPS referenced by the SPS.
[0489] When sps_video_parameter_set_id is equal to 0, the following applies:
[0490] – SPS does not refer to VPS.
[0491] – The VPS is not referenced when decoding each CLVS that references the SPS.
[0492] – The value of vps_max_layers_minus1 is inferred to be 0.
[0493] – A CVS shall contain only one layer (i.e. all VCL NAL units in a CVS shall have the same nuh_layer_id value).
[0494] –The value of GeneralLayerIdx[nuh_layer_id] is inferred to be 0.
[0495] – The value of vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is inferred to be 1.
[0496] When vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 1, the SPS referenced by a CLVS with a specific nuh_layer_id value (nuhLayerId) shall have nuh_layer_id equal to nuhLayerId.
[0497] chroma_format_idc indicates the chroma samples relative to the luma samples, as detailed in section 6.2.
[0498] separate_colour_plane_flag equal to 1 indicates that the 3 colour components in the 4:4:4 chroma format are coded separately. separate_colour_plane_flag equal to 0 indicates that the colour components are not coded separately. When separate_colour_plane_flag is not present, separate_colour_plane_flag is inferred to be 0. When separate_colour_plane_flag is equal to 1, the coded picture consists of 3 separate components, each consisting of coded samples of one colour plane (Y, Cb or Cr) and using the black and white coding syntax. In this case, each colour plane is associated with a specific colour_plane_id value.
[0499] NOTE 1: There is no dependency between colour planes with the same colour_plane_id value during the decoding process. For example, the decoding process of a black and white picture with one colour_plane_id value does not use any data of a black and white picture with a different colour_plane_id value for inter prediction.
[0500] Depending on the value of separate_colour_plane_flag, the variable ChromaArrayType is assigned the following values:
[0501] – If separate_colour_plane_flag is equal to 0, ChromaArrayType is set to chroma_format_idc.
[0502] – Otherwise (separate_colour_plane_flag is equal to 1), ChromaArrayType is set to 0.
[0503] The meaning of chroma_format_idc and separate_colour_plane_flag is used to indicate the chroma format:
[0504] chroma_format_idc separate_colour_plane_flag Chroma Format SubWidthC SubHeightC 0 0 Black and White 1 1 1 0 4:2:0 2 2 2 0 4:2:2 2 1 3 0 4:4:4 1 1 3 1 4:4:4 1 1
[0505] sps_poc_msb_flag equal to 1 indicates that the syntax element ph_poc_msb_cycle_present_flag is present in the PH referencing the SPS. sps_poc_msb_flag equal to 0 indicates that the syntax element ph_poc_msb_cycle_present_flag is not present in the PH referencing the SPS.
[0506] poc_msb_len_minus1+1 indicates the length in bits of the syntax element poc_msb_val when it is present in the PH of the reference SPS. The value range of poc_msb_len_minus1 shall be 0 to 32–log2_max_pic_order_cnt_lsb_minus4–5 (inclusive).
[0507]
[0508] ph_poc_msb_present_flag equal to 1 indicates that the syntax element poc_msb_val is present in the PH. ph_poc_msb_present_flag equal to 0 indicates that the syntax element poc_msb_val is not present in the PH. The value of ph_poc_msb_present_flag shall be equal to 0 when vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 0 and there is a picture in the current AU in the reference layer of the current layer.
[0509] poc_msb_val represents the POC MSB value of the current picture. The length of the syntax element poc_msb_val is (poc_msb_len_minus1+1) bits.
[0510] The following is the derivation of the picture order count (POC) of the current image.
[0511] Decoding process for image sequence numbering
[0512] The output of this process is PicOrderCntVal, which is the image order number of the current image.
[0513] Each coded picture is associated with a picture order number variable (denoted PicOrderCntVal).
[0514] If vps_independent_layer_flag[GeneralLayerIdx[nuh_layer_id]] is equal to 0 and there is a picture picA in the current AU in the reference layer of the current layer, then PicOrderCntVal is derived as PicOrderCntVal of picA, and all VCL NAL units of the current AU shall have the same slice_pic_order_cnt_lsb value. Otherwise, PicOrderCntVal of the current picture is derived as follows.
[0515] When ph_poc_msb_present_flag is equal to 0 and the current picture is not a CLVSS picture, the variables prevPicOrderCntLsb and prevPicOrderCntMsb are derived as follows:
[0516] – Assume that prevTid0Pic is the previous picture in decoding order, where nuh_layer_id of the previous picture is equal to nuh_layer_id of the current picture, TemporalId of the previous picture is equal to 0, and the previous picture is not a RASL picture or a RADL picture.
[0517] – The variable prevPicOrderCntLsb is set to slice_pic_order_cnt_lsb of prevTid0Pic.
[0518] –The variable prevPicOrderCntMsb is set to PicOrderCntMsb of prevTid0Pic.
[0519] The variable PicOrderCntMsb of the current image is derived as follows:
[0520] – If ph_poc_msb_present_flag is equal to 1, PicOrderCntMsb is set to poc_msb_val*MaxPicOrderCntLsb.
[0521] – Otherwise (ph_poc_msb_present_flag is equal to 0), if the current picture is a CLVSS picture, PicOrderCntMsb is set to 0.
[0522] – Otherwise, PicOrderCntMsb is derived as follows:
[0523]
[0524] PicOrderCntVal is derived as follows:
[0525] PicOrderCntVal=PicOrderCntMsb+slice_pic_order_cnt_lsb (197)
[0526] NOTE 1: Since PicOrderCntMsb is set to 0 for CLVSS pictures where poc_msb_val does not exist, PicOrderCntVal for these pictures is equal to slice_pic_order_cnt_lsb.
[0527] The value range of PicOrderCntVal should be –2 31 to 2 31 –1 (inclusive).
[0528] In one CVS, the PicOrderCntVal values of any two coded pictures with the same nuh_layer_id shall be different.
[0529] All pictures in any particular AU should have the same PicOrderCntVal value.
[0530] The function PicOrderCnt(picX) is as follows:
[0531] PicOrderCnt(picX) = PicOrderCntVal of image picX (198)
[0532] The function DiffPicOrderCnt(picA,picB) is as follows:
[0533] DiffPicOrderCnt(picA,picB)=PicOrderCnt(picA)–PicOrderCnt(picB)(199)
[0534] The code stream should not include the value range of DiffPicOrderCnt(picA,picB) used in the decoding process not being –2 15 to 2 15 –1 (including the end value).
[0535] Note 2: Assuming X is the current image, Y and Z are two other images in the same CVS, when DiffPicOrderCnt(X,Y) and DiffPicOrderCnt(X,Z) are both positive or both negative, Y and Z are considered to be in the same output order direction relative to X.
[0536] Constraints for sps_poc_msb_flag
[0537] The flag sps_poc_msb_flag is used to control whether the flag ph_poc_msb_cycle_present_flag will be present in the picture header. At the same time, mixing IRAP pictures and non-IRAP pictures within an AU only needs to be implemented in a multi-layer scenario. Therefore, in a single-layer coding scenario, ph_poc_msb_cycle_present_flag does not need to be present. Therefore, in a single-layer coding scenario, the value of sps_poc_msb_flag can be constrained to 0.
[0538] 1.2 Constraints of chroma_format_idc, separate_colour_plane_flag, bit_depth_minus8
[0539] The current motion compensation process can be used in inter-layer prediction, but cannot be used in scenarios where different layers have different formats (e.g. chroma_format_idc, separate_colour_plane_flag, bit_depth_minus8).
[0540] It is proposed to constrain the value of sps_poc_msb_flag to be equal to 0 in a multi-layer coding scenario.
[0541] It is proposed that constrained inter-layer prediction can only be used for scenarios where the current image in the current layer and the reference image in the reference layer have the same format.
[0542] The flag sps_poc_msb_flag is used to control whether the flag ph_poc_msb_cycle_present_flag will be present in the picture header. At the same time, mixing IRAP pictures and non-IRAP pictures within an AU only needs to be implemented in a multi-layer scenario. Therefore, in a single-layer coding scenario, ph_poc_msb_cycle_present_flag does not need to be present. Therefore, in a single-layer coding scenario, the value of sps_poc_msb_flag can be constrained to 0.
[0543] First embodiment (constraints of sps_poc_msb_flag)
[0544] The semantics of sps_poc_msb_flag can be modified as follows:
[0545] sps_poc_msb_flag equal to 1 indicates that the syntax element ph_poc_msb_cycle_present_flag is present in the PH of the referenced SPS. sps_poc_msb_flag equal to 0 indicates that the syntax element ph_poc_msb_cycle_present_flag is not present in the PH of the referenced SPS. When vps_max_layers_minus1 is equal to 0, the value of sps_poc_msb_flag shall be equal to 0.
[0546] Second Embodiment (Format Constraints)
[0547] The following constraints should be added to the specification:
[0548] vps_direct_ref_layer_flag[i][j] equal to 0 indicates that the layer with index j is not the direct reference layer of the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 indicates that the layer with index j is the direct reference layer of the layer with index i. When vps_direct_ref_layer_flag[i][j] (where i and j range from 0 to vps_max_layers_minus1, inclusive) is not present, vps_direct_ref_layer_flag[i][j] is inferred to be 0. When vps_independent_layer_flag[i] is equal to 0, at least one value of j shall be in the range 0 to i–1, inclusive, such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.
[0549] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] are derived as follows:
[0550]
[0551] The variable GeneralLayerIdx[i] represents the layer index of the layer whose nuh_layer_id is equal to vps_layer_id[i], and is derived as follows:
[0552] for(i=0;i<=vps_max_layers_minus1;i++) (38)
[0553] GeneralLayerIdx[vps_layer_id[i]]=i
[0554] Constraint Option A:
[0555] If the current layer is a dependent layer, the video in the current layer shall have the same chroma_format_idc as the video in the reference layer.
[0556] Furthermore, it can be said that inter-layer prediction can only be used in scenarios where the video in the current layer has the same chroma_format_idc as the video in the reference layer.
[0557] Constraint Option B:
[0558] If the current layer is a dependent layer, the video in the current layer shall have the same separate_colour_plane_flag as the video in the reference layer.
[0559] Furthermore, it can be said that inter-layer prediction can only be used for scenarios where the video in the current layer has the same separate_colour_plane_flag as the video in the reference layer.
[0560] Constraint Option C:
[0561] If the current layer is a dependent layer, the video in the current layer should have the same bit_depth_minus8 as the video in the reference layer.
[0562] Furthermore, it can be said that inter-layer prediction can only be used in scenarios where the video in the current layer has the same bit_depth_minus8 as the video in the reference layer.
[0563] Options A, B, and C can be combined:
[0564] For example, option A+B:
[0565] If the current layer is a dependent layer, the video in the current layer shall have the same chroma_format_idc, separate_colour_plane_flag as the video in the reference layer.
[0566] Furthermore, it can be said that inter-layer prediction can only be used in scenarios where the video in the current layer has the same chroma_format_idc, separate_colour_plane_flag as the video in the reference layer.
[0567] For example, option A+B+C:
[0568] If the current layer is a dependent layer, the video in the current layer shall have the same chroma_format_idc, separate_colour_plane_flag, bit_depth_minus8 as the video in the reference layer.
[0569] Furthermore, it can be said that inter-layer prediction can only be used in scenarios where the video in the current layer has the same chroma_format_idc, separate_colour_plane_flag, bit_depth_minus8 as the video in the reference layer.
[0570] Third embodiment (format constraints)
[0571] Constraints can also be added to other methods.
[0572] 8 Decoding process
[0573] 8.1 General Decoding Process
[0574] 8.1.1 Overview
[0575] The input of this process is the bitstream BitstreamToDecode. The output of this process is the decoded picture list.
[0576] The decoding process is specified so that all decoders conforming to a specified profile and level produce numerically identical cropped decoded output pictures when the decoding process associated with that profile is invoked for a codestream conforming to that profile and level. Any decoding process that produces the same cropped decoded output pictures (with the correct output order or output timing as specified) as the process described herein conforms to the decoding process requirements of this specification.
[0577] For each IRAP AU in the codestream, the following applies:
[0578] – If the AU is the first AU in decoding order in the codestream, each picture is an IDR picture, or each picture is the first picture in the decoding order after an EOS NAL unit in a layer, the variable NoIncorrectPicOutputFlag is set to 1.
[0579] – Otherwise, if an external module not specified in this specification is available to set the variable HandleCraAsCvsStartFlag to a value of AU, HandleCraAsCvsStartFlag is set to the value provided by the external module and NoIncorrectPicOutputFlag is set to HandleCraAsCvsStartFlag.
[0580] – Otherwise, HandleCraAsCvsStartFlag and NoIncorrectPicOutputFlag are both set to 0.
[0581] For each GDR AU in the codestream, the following applies:
[0582] – The variable NoIncorrectPicOutputFlag is set to 1 if the AU is the first AU in the decoding order in the codestream, or if each picture is the first picture in the decoding order after the EOS NAL unit in the layer.
[0583] – Otherwise, if an external module not detailed in this specification is available to set the variable HandleGdrAsCvsStartFlag to a value of AU, HandleGdrAsCvsStartFlag is set to the value provided by the external module and NoIncorrectPicOutputFlag is set to HandleGdrAsCvsStartFlag.
[0584] – Otherwise, HandleGdrAsCvsStartFlag and NoIncorrectPicOutputFlag are both set to 0.
[0585] Note: The above operations for IRAP images and GDR images are required to determine the CVS in the codestream.
[0586] The variable TargetOlsIdx (determines the OLS index of the target OLS to be decoded) and the variable Htid (determines the highest temporal sublayer to be decoded) are set by some external module not detailed in this specification. The bitstream BitstreamToDecode does not include any other layers than those included in the target OLS, nor does it include any NAL unit with a TemporalId greater than Htid.
[0587] Section 8.1.2 is called repeatedly for each coded picture in the decoding order in BitstreamToDecode.
[0588] Option A:
[0589] When BitstreamToDecode includes more than one layer, the following properties should be the same for each layer:
[0590] –chroma_format_idc
[0591] –separate_colour_plane_flag
[0592] Option B:
[0593] When BitstreamToDecode includes more than one layer, the following properties should be the same for each layer:
[0594] –bit_depth_minus8
[0595] Option C = Option A + Option B
[0596] When BitstreamToDecode includes more than one layer, the following properties should be the same for each layer:
[0597] –chroma_format_idc
[0598] –separate_colour_plane_flag
[0599] –bit_depth_minus8
[0600] (1) In single-layer coding scenarios, the value of sps_poc_msb_flag is constrained to be equal to 0 to ease the design of POC derivation.
[0601] (2) The formats of the current layer and reference layer for inter-layer prediction are constrained to simplify the design.
[0602] Figure 7 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Figure 7 As shown, the method for decoding a coded video code stream may include the following steps 701 to 703.
[0603] S701. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with an index of k is a direct reference layer of a layer with an index of i, and i and k are both integers and greater than or equal to 0.
[0604] Specifically, a layer is a sequence of images, the images in the image sequence share the same layer identifier or layer index.
[0605] Specifically, the device for decoding the encoded video code stream (eg Figure 3 The decoder shown in FIG. 1 is used to parse the code stream 21 (or generally referred to as the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantized coefficients 309 and / or the decoded encoding parameters ( Figure 3The entropy decoding unit 304 may be used to apply a decoding algorithm or scheme corresponding to the encoding scheme described in conjunction with the entropy encoding unit 270 in the encoder 20. The entropy decoding unit 304 may also be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units in the decoder 30. The video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or as an alternative to the slices and the corresponding syntax elements, the block groups and / or blocks and the corresponding syntax elements may be received and / or used.
[0606] In one embodiment, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS applies to the layer with index j and the layer with index i.
[0607] In one embodiment, the reference layer syntax element may be the syntax element vps_direct_ref_layer_flag[i][j] in the VPS table referenced above.
[0608] vps_direct_ref_layer_flag[i][j] equal to 0 indicates that the layer with index j is not the direct reference layer of the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 indicates that the layer with index j is the direct reference layer of the layer with index i. When vps_direct_ref_layer_flag[i][j] (where i and j range from 0 to vps_max_layers_minus1, inclusive) is not present, vps_direct_ref_layer_flag[i][j] is inferred to be 0. When vps_independent_layer_flag[i] is equal to 0, at least one value of j shall be in the range 0 to i–1, inclusive, such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.
[0609] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] are derived as follows:
[0610]
[0611] Specifically, vps_direct_ref_layer_flag[i][j] equal to 1 indicates that the layer with index j is the direct reference layer of the layer with index i.
[0612] S702. Determine whether the layer with index j is the reference layer of the layer with index i according to the value of the reference layer syntax element, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0613] In one embodiment, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0614] Specifically, regarding the reference layer, there are two scenarios: one scenario is that the layer indexed by A is the direct reference layer of the layer indexed by B; the other scenario is that when the layer indexed by C is the reference layer of the layer indexed by B, the layer indexed by A is the reference layer of the layer indexed by C, and the layer indexed by A is not the direct reference layer of the layer indexed by B, the layer indexed by A is the indirect reference layer of the layer indexed by B. The reference layer includes a direct reference layer or an indirect reference layer. The layer indexed by A being the direct reference layer of the layer indexed by B means that the layer indexed by A includes at least one reference image of the image in the layer indexed by B. In step S702, the layer indexed by j is the reference layer of the layer indexed by k; if the value of the reference layer syntax element indicates that the layer indexed by k is the direct reference layer of the layer indexed by i, the layer indexed by j is the reference layer of the layer indexed by i.
[0615] S703. If a condition is met, predict the image in the layer with index i according to the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0616] Specifically, if the layer with index j is the reference layer of the layer with index i, the image in the layer with index i can be predicted based on the layer with index j, and the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j.
[0617] Specifically, the device for decoding the encoded video code stream (eg Figure 3 The decoder 30 shown is used to use inter-frame prediction for the image in the layer indexed by i according to the layer indexed by j, and generate a prediction block 365 for the video block of the current video slice according to the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-frame prediction, the prediction block can be generated according to one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar processes can be applied to or by embodiments using block groups (e.g., video block groups) and / or blocks (e.g., video blocks). For example, the video can be decoded using I, P or B block groups and / or blocks.
[0618] Specifically, inter-layer prediction refers to using inter-frame prediction for an image in a layer with an index of i based on a reference image in a layer with an index of j.
[0619] In one embodiment, the chroma format related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0620] Specifically, the SPS is applied to a sequence of images, and a layer is a sequence of images, and the SPS is applied to a layer indexed by j or a layer indexed by i.
[0621] In one embodiment, the chroma format related syntax element may be the syntax element chroma_format_idc in the SPS table cited above.
[0622] chroma_format_idc indicates the chroma samples relative to the luma samples, as detailed in section 6.2.
[0623] The meaning of chroma_format_idc and separate_colour_plane_flag is used to indicate the chroma format:
[0624] Table: SubWidthC and SubHeightC derived from chroma_format_idc
[0625] chroma_format_idc separate_colour_plane_flag Chroma Format SubWidthC SubHeightC 0 0 Black and White 1 1 1 0 4:2:0 2 2 2 0 4:2:2 2 1 3 0 4:4:4 1 1 3 1 4:4:4 1 1
[0626] According to an embodiment of the present application, a reference layer syntax element is obtained by parsing the coded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index k is the direct reference layer of the layer with index i, i and k are both integers and greater than or equal to 0; according to the value of the reference layer syntax element, it is determined whether the layer with index j is the reference layer of the layer with index i, wherein the layer with index j is the reference layer of the layer with index k, j is an integer and greater than or equal to 0; if the condition is satisfied, the image in the layer with index i is predicted according to the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is the reference layer of the layer with index i. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0627] Figure 8 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Figure 8 As shown, the method for decoding a coded video code stream may include the following steps 801 to 803.
[0628] S801. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with index j is a direct reference layer of a layer with index i, and i and j are both integers and greater than or equal to 0.
[0629] S802. If a condition is met, predict the image in the layer with index i based on the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is the direct reference layer of the layer with index i.
[0630] It should be noted that if the value of the reference layer syntax element indicates that the layer with index j is not the direct reference layer of the layer with index i, then the layer with index j can be the reference layer, that is, the layer with index j can be the indirect reference layer of the layer with index i, or it can be the reference layer.
[0631] According to an embodiment of the present application, a reference layer syntax element is obtained by parsing the coded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index j is a direct reference layer of the layer with index i, i and j are both integers and greater than or equal to 0; if the condition is met, the image in the layer with index i is predicted according to the layer with index j, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0632] according to Figure 7 and Figure 8 In the illustrated embodiment, the method for decoding the encoded video stream further includes:
[0633] The chroma format related syntax elements applied to the layer with index i and the chroma format related syntax elements applied to the layer with index j are obtained by parsing the encoded video code stream; the condition also includes that the value of the chroma format related syntax elements applied to the layer with index i is the same as the value of the chroma format related syntax elements applied to the layer with index j.
[0634] Specifically, when a layer with index j is a reference layer of a layer with index i, and the value of a chroma format-related syntax element applied to the layer with index i is the same as the value of a chroma format-related syntax element applied to the layer with index j, the image in the layer with index i can only be predicted based on the layer with index j.
[0635] Fig. 9 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Fig. 9 As shown, according to Figure 7 The method shown in the figure, the method for decoding the encoded video code stream may include the following steps 901 to 906.
[0636] S901. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with an index of k is a direct reference layer of a layer with an index of i, and i and k are both integers and are greater than or equal to 0.
[0637] S902. Determine, according to the value of the reference layer syntax element, whether the layer with index j is the reference layer of the layer with index i, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0638] S903. If a condition is met, predict the image in the layer with index i according to the layer with index j, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0639] S904. If the layer with index j is the reference layer of the layer with index i, and the value of the chroma format-related syntax element applied to the layer with index i is different from the value of the chroma format-related syntax element applied to the layer with index j, stop decoding the encoded video stream.
[0640] S905: If the layer with index j is not a reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0641] Specifically, when the layer with index j is not the direct reference layer of the layer with index i and the layer with index k cannot be found after traversing all layers, the layer with index k is the direct reference layer of the layer with index i, and the layer with index j is the reference layer of the layer with index k, it is determined that the layer with index j is not the reference layer of the layer with index i.
[0642] In one embodiment, if the layer with index j is not the direct reference layer of the layer with index i, determine whether the layer with index j is an indirect reference layer of the layer with index i; if the layer with index j is an indirect reference layer of the layer with index i, use the layer with index j to predict the image in the layer with index i; if the layer with index j is not the indirect reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0643] When it is determined by traversing all layers that a layer with index k is a direct reference layer of a layer with index i and a layer with index j is a reference layer of the layer with index k, it is determined that the layer with index j is an indirect reference layer of the layer with index i.
[0644] S906. If the condition is met, without obtaining the chroma format-related syntax element applied to the layer with index i by parsing the encoded video code stream, determine that the value of the chroma format-related syntax element applied to the layer with index i is the value of the chroma format-related syntax element applied to the layer with index j.
[0645] Specifically, when a layer with an index of j is a reference layer of a layer with an index of i, without obtaining the chroma format-related syntax elements of the layer with an index of i by parsing the encoded video code stream, it is determined that the value of the chroma format-related syntax elements applied to the layer with an index of i is the value of the chroma format-related syntax elements applied to the layer with an index of j.
[0646] According to an embodiment of the present application, if the layer with index j is the reference layer of the layer with index i, and the value of the chroma format related syntax element applied to the layer with index i is different from the value of the chroma format related syntax element applied to the layer with index j, stop decoding the coded video code stream; if the layer with index j is not the reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i; if the condition is met, without obtaining the chroma format related syntax element applied to the layer with index i by parsing the coded video code stream, determine that the value of the chroma format related syntax element applied to the layer with index i is the value of the chroma format related syntax element applied to the layer with index j. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0647] Fig.10 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Fig.10 As shown, the method for decoding a coded video code stream may include the following steps 1001 to 1003.
[0648] S1001. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with an index of k is a direct reference layer of a layer with an index of i, and i and k are both integers and greater than or equal to 0.
[0649] In one embodiment, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS applies to the layer with index j and the layer with index i.
[0650] S1002. Determine whether the layer with index j is the reference layer of the layer with index i according to the value of the reference layer syntax element, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0651] In one embodiment, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0652] S1003. If a condition is met, predict the image in the layer with index i based on the layer with index j, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0653] In one embodiment, the bit-depth related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit-depth related syntax element applies.
[0654] In one embodiment, the bit depth related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0655] In one embodiment, the bit depth related syntax element may be the syntax element bit_depth_minus8 in the SPS table referenced above.
[0656] bit_depth_minus8 represents the bit depth BitDepth of samples in the luma and chroma arrays and the value of the luma and chroma quantization parameter range offset QpBdOffset as follows:
[0657] BitDepth=8+bit_depth_minus8 (45)
[0658] QpBdOffset=6*bit_depth_minus8 (46)
[0659] bit_depth_minus8 should range from 0 to 8 (inclusive).
[0660] The principle of this embodiment is Figure 7 The embodiments shown are similar and will not be described again for the sake of brevity.
[0661] According to an embodiment of the present application, a reference layer syntax element is obtained by parsing the coded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index k is a direct reference layer of the layer with index i, i and k are both integers and greater than or equal to 0; according to the value of the reference layer syntax element, it is determined whether the layer with index j is a reference layer of the layer with index i, wherein the layer with index j is a reference layer of the layer with index k, j is an integer and greater than or equal to 0; if the condition is satisfied, the image in the layer with index i is predicted according to the layer with index j, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i. The present invention implements the constraints of the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0662] Fig.11 A schematic flow chart of a method for decoding a coded video stream provided in one embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Fig.11 As shown, the method for decoding a coded video code stream may include the following steps 1101 to 1102.
[0663] S1101. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with index j is a direct reference layer of a layer with index i, and i and j are both integers and greater than or equal to 0.
[0664] S1102. If a condition is met, predict the image in the layer with index i based on the layer with index j, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i.
[0665] The principle of this embodiment is Figure 8 The embodiments shown are similar and will not be described again for the sake of brevity.
[0666] According to an embodiment of the present application, a reference layer syntax element is obtained by parsing the coded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index j is a direct reference layer of the layer with index i, i and j are both integers and greater than or equal to 0; if the condition is met, the image in the layer with index i is predicted according to the layer with index j, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0667] according to Fig.10 and Fig.11 In the illustrated embodiment, the method for decoding the encoded video stream further includes:
[0668] The bit depth-related syntax element applied to the layer with index i and the bit depth-related syntax element applied to the layer with index j are obtained by parsing the encoded video code stream; the condition also includes that a value of the bit depth-related syntax element applied to the layer with index i is the same as a value of the bit depth-related syntax element applied to the layer with index j.
[0669] Fig.12 1 is a schematic flow chart of a method for decoding a coded video stream provided by an embodiment of the present application. The method may be performed by a device for decoding a coded video stream. Fig.12 As shown, according to Fig.10 The method shown in the figure, the method for decoding the encoded video code stream may include the following steps 1201 to 1206.
[0670] S1201. Obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether a layer with an index of k is a direct reference layer of a layer with an index of i, and i and k are both integers and greater than or equal to 0.
[0671] S1202. Determine whether the layer with index j is the reference layer of the layer with index i according to the value of the reference layer syntax element, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0672] S1203. If a condition is met, predict the image in the layer with index i according to the layer with index j, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0673] S1204. If the layer with index j is the reference layer of the layer with index i, and the value of the bit depth-related syntax element applied to the layer with index i is different from the value of the bit depth-related syntax element applied to the layer with index j, stop decoding the encoded video stream.
[0674] S1205: If the layer with index j is not a reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0675] S1206. If the condition is met, without obtaining the bit depth-related syntax element applied to the layer with index i by parsing the coded video code stream, determine that the value of the bit depth-related syntax element applied to the layer with index i is the value of the bit depth-related syntax element applied to the layer with index j.
[0676] The principle of this embodiment is Fig. 9 The embodiments shown are similar and will not be described again for the sake of brevity.
[0677] According to an embodiment of the present application, if the layer with index j is the reference layer of the layer with index i, and the value of the bit depth related syntax element applied to the layer with index i is different from the value of the bit depth related syntax element applied to the layer with index j, stop decoding the coded video code stream; if the layer with index j is not the reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i; if the condition is satisfied, without obtaining the bit depth related syntax element applied to the layer with index i by parsing the coded video code stream, determine that the value of the bit depth related syntax element applied to the layer with index i is the value of the bit depth related syntax element applied to the layer with index j. The present invention implements the constraints of the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0678] Fig.13 1 is a schematic flow chart of a method for encoding a video provided by an embodiment of the present application. The method may be performed by a device for encoding a video. Fig.13As shown, the method for encoding a video may include the following steps 1301 and 1302.
[0679] S1301. Determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0680] The apparatus for encoding a video (eg Figure 2 The encoder 20 shown can, for example, be used to determine whether a layer indexed by j is a direct reference layer of a layer indexed by i, and to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to a motion estimation unit.
[0681] S1302. If the layer with index j is the direct reference layer of the layer with index i, encode into the video bitstream a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i, and encode into the video bitstream a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j.
[0682] In one embodiment, the reference layer syntax element may be the syntax element vps_direct_ref_layer_flag[i][j] in the VPS table referenced above.
[0683] vps_direct_ref_layer_flag[i][j] equal to 0 indicates that the layer with index j is not the direct reference layer of the layer with index i. vps_direct_ref_layer_flag[i][j] equal to 1 indicates that the layer with index j is the direct reference layer of the layer with index i. When vps_direct_ref_layer_flag[i][j] (where i and j range from 0 to vps_max_layers_minus1, inclusive) is not present, vps_direct_ref_layer_flag[i][j] is inferred to be 0. When vps_independent_layer_flag[i] is equal to 0, at least one value of j shall be in the range 0 to i–1, inclusive, such that the value of vps_direct_ref_layer_flag[i][j] is equal to 1.
[0684] The variables NumDirectRefLayers[i], DirectRefLayerIdx[i][d], NumRefLayers[i], RefLayerIdx[i][r], and LayerUsedAsRefLayerFlag[j] are derived as follows:
[0685]
[0686] In one embodiment, the chroma format related syntax element may be the syntax element chroma_format_idc in the SPS table cited above.
[0687] chroma_format_idc indicates the chroma samples relative to the luma samples, as detailed in section 6.2.
[0688] The meaning of chroma_format_idc and separate_colour_plane_flag is used to indicate the chroma format:
[0689] Table: SubWidthC and SubHeightC derived from chroma_format_idc
[0690] chroma_format_idc separate_colour_plane_flag Chroma Format SubWidthC SubHeightC 0 0 Black and White 1 1 1 0 4:2:0 2 2 2 0 4:2:2 2 1 3 0 4:4:4 1 1 3 1 4:4:4 1 1
[0691] Specifically, the apparatus for encoding a video (eg Figure 2The encoder 20 in the embodiment is used to: encode into a video bitstream a reference layer syntax element whose value indicates that the layer with index j is a direct reference layer of the layer with index i; encode into the video bitstream a syntax element related to the chroma format applied to the layer with index i and a syntax element related to the chroma format applied to the layer with index j; encode into the video bitstream an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CAVLC), an arithmetic coding scheme, binarization, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (probability interval partitioning entropy, PIPE) encoding or other entropy encoding methods or techniques) are applied or not applied (without compression) to the quantized coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters and / or other syntax elements to obtain the encoded image data 21 that can be output through the output terminal 272 in the form of the encoded code stream 21, so that a device for decoding the encoded code stream (e.g. Figure 3 The video decoder 30 shown can receive and use these parameters for decoding. The encoded bitstream 21 can be sent to the video decoder 30, or stored in a memory for later transmission or retrieval by the video decoder 30.
[0692] According to an embodiment of the present application, it is determined whether the layer with index j is the direct reference layer of the layer with index i, wherein i and j are both integers and are greater than or equal to 0; if the layer with index j is the direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i is encoded into a video bitstream, and a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0693] Fig.141 is a schematic flow chart of a method for encoding a video provided by an embodiment of the present application. The method may be performed by a device for encoding a video. Fig.14 As shown, according to Fig.13 The method shown in the figure, the method for encoding a video may include the following steps 1401 to 1404.
[0694] S1401. Determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0695] S1402. If the layer with index j is the direct reference layer of the layer with index i, encode into the video bitstream a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i, and encode into the video bitstream a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j.
[0696] S1403: If the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0697] The set of (possible) inter-frame prediction modes depends on the available reference image (i.e., at least part of the decoded image stored in the DPB as mentioned above) and other inter-frame prediction parameters, for example, on whether the entire reference image or only a part of the reference image (e.g. a search window area around the area of the current block) is used to search for the best matching reference block, and / or on, for example, whether pixel interpolation is applied, such as half-pixel, quarter-pixel and / or 1 / 16-pixel interpolation.
[0698] S1404: If the layer with index j is not a reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0699] In one embodiment, if the layer with index j is not the direct reference layer of the layer with index i, determine whether the layer with index j is an indirect reference layer of the layer with index i; if the layer with index j is an indirect reference layer of the layer with index i, use the layer with index j to predict the image in the layer with index i; if the layer with index j is not the indirect reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0700] According to an embodiment of the present application, it is determined whether the layer with index j is the direct reference layer of the layer with index i, wherein i and j are both integers and greater than or equal to 0; if the layer with index j is the direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i is encoded into a video bitstream, and a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the value of the chroma format-related syntax element applied to the layer with index i is the same as the value of the chroma format-related syntax element applied to the layer with index j; if the layer with index j is the direct reference layer of the layer with index i, an image in the layer with index i is predicted according to the layer with index j. The present invention implements the constraints of the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0701] Fig.15 1 is a schematic flow chart of a method for encoding a video provided by an embodiment of the present application. The method may be performed by a device for encoding a video. Fig.15 As shown, the method for encoding a video may include the following steps 1501 and 1502.
[0702] S1501. Determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0703] S1502. If the layer with index j is the direct reference layer of the layer with index i, encode into the video bitstream a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i, and encode into the video bitstream a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0704] In one embodiment, the bit-depth related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit-depth related syntax element applies.
[0705] In one embodiment, the bit depth related syntax element may be the syntax element bit_depth_minus8 in the SPS table referenced above.
[0706] bit_depth_minus8 represents the bit depth BitDepth of samples in the luma and chroma arrays and the value of the luma and chroma quantization parameter range offset QpBdOffset as follows:
[0707] BitDepth=8+bit_depth_minus8 (45)
[0708] QpBdOffset=6*bit_depth_minus8 (46)
[0709] bit_depth_minus8 should range from 0 to 8 (inclusive).
[0710] The principle of this embodiment is Fig.13 The embodiments shown are similar and will not be described again for the sake of brevity.
[0711] According to an embodiment of the present application, in one embodiment, the bit depth related syntax element may be the syntax element bit_depth_minus8 in the SPS table cited above.
[0712] bit_depth_minus8 represents the bit depth BitDepth of samples in the luma and chroma arrays and the value of the luma and chroma quantization parameter range offset QpBdOffset as follows:
[0713] BitDepth=8+bit_depth_minus8 (45)
[0714] QpBdOffset=6*bit_depth_minus8 (46)
[0715] bit_depth_minus8 should range from 0 to 8 (inclusive).
[0716] The principle of this embodiment is Figure 7 The embodiments shown are similar and will not be described again for the sake of brevity.
[0717] According to an embodiment of the present application, it is determined whether a layer with index j is a direct reference layer of a layer with index i, wherein i and j are both integers and are greater than or equal to 0; if the layer with index j is a direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is a direct reference layer of the layer with index i is encoded into a video bitstream, and a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j. The present invention implements the constraints on the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0718] Fig.16 1 is a schematic flow chart of a method for encoding a video provided by an embodiment of the present application. The method may be performed by a device for encoding a video. Fig.16 As shown, according to Fig.15 The method shown in the figure, the method for encoding a video may include the following steps 1601 to 1604.
[0719] S1601. Determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0720] S1602. If the layer with index j is the direct reference layer of the layer with index i, encode into the video bitstream a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i, and encode into the video bitstream a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0721] S1603: If the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0722] S1604: If the layer with index j is not a reference layer of the layer with index i, do not use the layer with index j to predict the image in the layer with index i.
[0723] According to an embodiment of the present application, it is determined whether a layer with index j is a direct reference layer of a layer with index i, wherein i and j are both integers and are greater than or equal to 0; if the layer with index j is a direct reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is a direct reference layer of the layer with index i is encoded into a video bitstream, and a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j; if the layer with index j is a direct reference layer of the layer with index i, an image in the layer with index i is predicted according to the layer with index j. The present invention implements the constraints of the formats of the current layer and the reference layer for inter-layer prediction, thereby simplifying the design.
[0724] Fig.17 FIG. 1 is a structural diagram of a device for decoding a coded video stream provided by an embodiment of the present application. Fig.17 As shown, the apparatus for decoding a coded video code stream may include an acquisition unit 1701 , a determination unit 1702 and a prediction unit 1703 .
[0725] The acquisition unit 1701 is used to obtain a reference layer syntax element by parsing the encoded video code stream, wherein the value of the reference layer syntax element indicates whether the layer with index k is a direct reference layer of the layer with index i, and i and k are both integers and greater than or equal to 0.
[0726] Determination unit 1702 is used to determine whether the layer with index j is the reference layer of the layer with index i according to the value of the reference layer syntax element, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0727] The prediction unit 1703 is used to: predict the image in the layer with index i according to the layer with index j if the condition is met, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0728] In one embodiment, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0729] In one embodiment, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS applies to the layer with index j and the layer with index i.
[0730] In one embodiment, the chroma format related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0731] The acquisition unit 1701 may be or include Figure 3 In the entropy decoding unit 304 of the decoder 30, the determination unit 1702 may be or include Figure 3 In the mode application unit 360 of the decoder 30, the prediction unit 1703 may be or include Figure 3 In the inter-frame prediction unit 344 of the decoder 30.
[0732] It should be noted that the acquisition unit 1701, the determination unit 1702 and the prediction unit 1703 may be software modules or hardware circuits.
[0733] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Figure 7 The method in Figure 7 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig.17 The structures of the devices for decoding the coded video code stream shown are the same and will not be described again for the sake of brevity.
[0734] Fig.18 FIG. 1 is a structural diagram of a device for decoding a coded video stream provided by an embodiment of the present application. Fig.18 As shown, the apparatus for decoding a coded video code stream may include an acquisition unit 1801 and a prediction unit 1802 .
[0735] The acquisition unit 1801 is used to obtain a reference layer syntax element by parsing the encoded video code stream, wherein the value of the reference layer syntax element indicates whether the layer with index j is a direct reference layer of the layer with index i, and i and j are both integers and greater than or equal to 0.
[0736] The prediction unit 1802 is used to: predict the image in the layer with index i according to the layer with index j if the condition is met, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is the direct reference layer of the layer with index i.
[0737] The acquisition unit 1801 may be or include Figure 3 In the entropy decoding unit 304 of the decoder 30 in, the prediction unit 1802 may be or include Figure 3 In the inter-frame prediction unit 344 of the decoder 30.
[0738] It should be noted that the acquisition unit 1801 and the prediction unit 1802 may be software modules or hardware circuits.
[0739] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Figure 8 The method in Figure 8 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig.18 The apparatus for decoding the encoded video code stream shown is the same and will not be described again for the sake of brevity.
[0740] according to Fig.17 and Fig.18 In the embodiment shown, the acquisition unit 1701 or the acquisition unit 1801 is further used to obtain the chroma format-related syntax elements applied to the layer with index i and the chroma format-related syntax elements applied to the layer with index j by parsing the encoded video code stream; the condition also includes that the value of the chroma format-related syntax elements applied to the layer with index i is the same as the value of the chroma format-related syntax elements applied to the layer with index j.
[0741] Fig.19 FIG. 1 is a structural diagram of a device for decoding a coded video stream provided by an embodiment of the present application. Fig.19 As shown, according to Fig.17 As shown in the device, the device for decoding a coded video code stream may include an acquisition unit 1701, a determination unit 1702, a prediction unit 1703 and a stop unit 1704.
[0742] The stopping unit 1704 is used to: stop decoding the encoded video code stream if the layer with index j is the reference layer of the layer with index i, and the value of the chroma format related syntax element applied to the layer with index i is different from the value of the chroma format related syntax element applied to the layer with index j.
[0743] The prediction unit 1703 is further configured to: if the layer with index j is not a reference layer of the layer with index i, not use the layer with index j to predict the image in the layer with index i.
[0744] The determination unit 1702 is also used to: if the condition is met, without obtaining the chroma format related syntax element applied to the layer with index i by parsing the encoded video code stream, determine that the value of the chroma format related syntax element applied to the layer with index i is the value of the chroma format related syntax element applied to the layer with index j.
[0745] The stopping unit 1704 in this embodiment may also be or include Figure 3 In the entropy decoding unit 304 of the decoder 30.
[0746] It should be noted that the acquisition unit 1701 , the determination unit 1702 , the prediction unit 1703 and the stopping unit 1704 may be software modules or hardware circuits.
[0747] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Fig. 9 The method in Fig. 9 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig.19 The apparatus for decoding the encoded video code stream shown is the same and will not be described again for the sake of brevity.
[0748] Fig. 20 FIG. 1 is a structural diagram of a device for decoding a coded video stream provided by an embodiment of the present application. Fig. 20 As shown, the apparatus for decoding a coded video code stream may include an acquisition unit 2001 , a determination unit 2002 and a prediction unit 2003 .
[0749] The acquisition unit 2001 is used to obtain a reference layer syntax element by parsing the encoded video code stream, wherein the value of the reference layer syntax element indicates whether the layer with index k is a direct reference layer of the layer with index i, and i and k are both integers and greater than or equal to 0.
[0750] Determination unit 2002 is used to determine whether the layer with index j is the reference layer of the layer with index i according to the value of the reference layer syntax element, wherein the layer with index j is the reference layer of the layer with index k, and j is an integer greater than or equal to 0.
[0751] The prediction unit 2003 is used to: predict the image in the layer with index i according to the layer with index j if a condition is met, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the layer with index j is a reference layer of the layer with index i.
[0752] In one embodiment, if the value of the reference layer syntax element indicates that the layer with index k is the direct reference layer of the layer with index i, then the layer with index j is the reference layer of the layer with index i.
[0753] In one embodiment, the bit-depth related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit-depth related syntax element applies.
[0754] In one embodiment, the reference layer syntax element is a video parameter set (VPS) level syntax element, wherein the VPS applies to the layer with index j and the layer with index i.
[0755] In one embodiment, the bit depth related syntax element is a sequence parameter set (SPS) level syntax element, wherein the SPS is applied to the layer with index j or the layer with index i.
[0756] The acquisition unit 2001 may be or include Figure 3 In the entropy decoding unit 304 of the decoder 30, the determination unit 2002 may be or include Figure 3 In the mode application unit 360 of the decoder 30, the prediction unit 2003 may be or include Figure 3 In the inter-frame prediction unit 344 of the decoder 30.
[0757] It should be noted that the acquisition unit 2001, the determination unit 2002 and the prediction unit 2003 may be software modules or hardware circuits.
[0758] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Fig.10 The method in Fig.10 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig. 20The apparatus for decoding the encoded video code stream shown is the same and will not be described again for the sake of brevity.
[0759] Fig.21 FIG. 1 is a structural diagram of a device for decoding a coded video stream provided by an embodiment of the present application. Fig.21 As shown, the apparatus for decoding a coded video code stream may include an acquisition unit 2101 and a prediction unit 2102 .
[0760] The acquisition unit 2101 is used to obtain a reference layer syntax element by parsing the encoded video bitstream, wherein the value of the reference layer syntax element indicates whether the layer with index j is a direct reference layer of the layer with index i, and i and j are both integers and greater than or equal to 0.
[0761] The prediction unit 2102 is used to: predict the image in the layer with index i according to the layer with index j if the condition is met, wherein the value of the bit depth related syntax element applied to the layer with index i is the same as the value of the bit depth related syntax element applied to the layer with index j, and the condition includes that the value of the reference layer syntax element indicates that the layer with index j is the direct reference layer of the layer with index i.
[0762] The acquisition unit 2101 may be or include Figure 3 In the entropy decoding unit 304 of the decoder 30, the prediction unit 2102 may be or include Figure 3 In the inter-frame prediction unit 344 of the decoder 30.
[0763] It should be noted that the acquisition unit 2101 and the prediction unit 2102 may be software modules or hardware circuits.
[0764] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Fig.11 The method in Fig.11 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig.21 The apparatus for decoding the encoded video code stream shown is the same and will not be described again for the sake of brevity.
[0765] according to Fig. 20 and Fig.21 In the embodiment shown, the acquisition unit 2001 or the acquisition unit 2101 is further used to obtain the bit depth-related syntax element applied to the layer with index i and the bit depth-related syntax element applied to the layer with index j by parsing the encoded video code stream; the condition also includes that the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0766] Fig. 22 The structure diagram of a device for decoding a coded video stream provided by one embodiment of the present application is shown in FIG. Fig. 22 As shown, according to Fig. 20 As shown in the device, the device for decoding a coded video code stream may include an acquisition unit 2001, a determination unit 2002, a prediction unit 2003 and a stop unit 2004.
[0767] The stopping unit 2004 is used to: stop decoding the encoded video code stream if the layer with index j is the reference layer of the layer with index i, and the value of the bit depth related syntax element applied to the layer with index i is different from the value of the bit depth related syntax element applied to the layer with index j.
[0768] The prediction unit 2003 is further configured to: if the layer with index j is not a reference layer of the layer with index i, not use the layer with index j to predict the image in the layer with index i.
[0769] The determination unit 2002 is further used to: determine that the value of the bit depth-related syntax element applied to the layer with index i is the value of the bit depth-related syntax element applied to the layer with index j without obtaining the bit depth-related syntax element applied to the layer with index i by parsing the encoded video code stream.
[0770] The stop unit 2004 in this embodiment may also be or include Figure 3 In the entropy decoding unit 304 of the decoder 30.
[0771] It should be noted that the acquisition unit 2001, the determination unit 2002, the prediction unit 2003 and the stop unit 2004 may be software modules or hardware circuits.
[0772] The apparatus for decoding a coded video code stream in this embodiment can be used to perform Fig.12 The method in Fig.12 The structure of the device for decoding the coded video code stream involved in the method shown can also be the same as Fig. 22 The structures of the devices for decoding the coded video code stream shown are the same and will not be described again for the sake of brevity.
[0773] It should be noted that the execution Figures 7 to 12 The method of the device or Figures 17 to 22 The device for decoding the coded video code stream may be or include Figure 1A Destination device 14, Figure 1B The video decoder 30 in Figure 3Decoder 30 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0774] Fig.23 The structure diagram of a device for encoding a video provided by one embodiment of the present application is shown in FIG. Fig.23 As shown, the apparatus for encoding a video may include a determining unit 2301 and an encoding unit 2302 .
[0775] The determination unit 2301 is used to determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0776] The encoding unit 2302 is used to: if the layer with index j is the direct reference layer of the layer with index i, encode the reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i into the video bitstream, and encode the chroma format related syntax element applied to the layer with index i and the chroma format related syntax element applied to the layer with index j into the video bitstream, wherein the value of the chroma format related syntax element applied to the layer with index i is the same as the value of the chroma format related syntax element applied to the layer with index j.
[0777] The determining unit 2301 may be or include Figure 2 In the mode selection unit 260 of the decoder 20, the encoding unit 2302 may be or include Figure 2 In the entropy coding unit 270 of the encoder 20.
[0778] It should be noted that the determination unit 2301 and the encoding unit 2302 may be software modules or hardware circuits.
[0779] The apparatus for encoding a video in this embodiment can be used to perform Fig.13 The method in Fig.13 The structure of the apparatus for encoding a video involved in the method shown can also be the same as Fig.23 The apparatus for encoding the video shown is the same and will not be described again for the sake of brevity.
[0780] Fig.24 The structure diagram of a device for encoding a video provided by one embodiment of the present application is shown in FIG. Fig.24 As shown, according to Fig.23 The device shown in the figure may further include a first prediction unit 2303 and a second prediction unit 2304.
[0781] The first prediction unit 2303 is used to: if the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0782] The second prediction unit 2304 is used for: if the layer with index j is not a reference layer of the layer with index i, not using the layer with index j to predict the image in the layer with index i.
[0783] The first prediction unit 2303 and the second prediction unit 2304 may be or include Figure 2 In the inter-frame prediction unit 244 of the decoder 20.
[0784] It should be noted that the determination unit 2301, the encoding unit 2302, the first prediction unit 2303 and the second prediction unit 2304 may be software modules or hardware circuits.
[0785] The apparatus for encoding a video in this embodiment can be used to perform Fig.14 The method in Fig.14 The structure of the apparatus for encoding a video involved in the method shown can also be the same as Fig.24 The apparatus for encoding the video shown is the same and will not be described again for the sake of brevity.
[0786] Fig.25 The structure diagram of a device for encoding a video provided by one embodiment of the present application is shown in FIG. Fig.25 As shown, the apparatus for encoding a video may include a determining unit 2501 and an encoding unit 2502 .
[0787] The determination unit 2501 is used to determine whether a layer with an index of j is a direct reference layer of a layer with an index of i, where i and j are both integers and are greater than or equal to 0.
[0788] The coding unit 2502 is used to: if the layer with index j is the direct reference layer of the layer with index i, encode a reference layer syntax element whose value indicates that the layer with index j is the direct reference layer of the layer with index i into the video bitstream, and encode a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j into the video bitstream, wherein the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j.
[0789] In one embodiment, the bit-depth related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit-depth related syntax element applies.
[0790] The determining unit 2501 may be or include Figure 2 In the mode selection unit 260 of the decoder 20, the encoding unit 2502 may be or include Figure 2 In the entropy coding unit 270 of the encoder 20.
[0791] It should be noted that the determination unit 2501 and the encoding unit 2502 can be software modules or hardware circuits.
[0792] The apparatus for encoding a video in this embodiment can be used to perform Fig.15 The method in Fig.15 The structure of the apparatus for encoding a video involved in the method shown can also be the same as Fig.25 The apparatus for encoding the video shown is the same and will not be described again for the sake of brevity.
[0793] Fig.26 The structure diagram of a device for encoding a video provided by one embodiment of the present application is shown in FIG. Fig.26 As shown, according to Fig.25 The device shown in the figure may further include a first prediction unit 2503 and a second prediction unit 2504.
[0794] The first prediction unit 2503 is used to: if the layer with index j is a direct reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
[0795] The second prediction unit 2504 is used for: if the layer with index j is not a reference layer of the layer with index i, not using the layer with index j to predict the image in the layer with index i.
[0796] The first prediction unit 2503 and the second prediction unit 2504 may be or include Figure 2 In the inter-frame prediction unit 244 of the decoder 20.
[0797] It should be noted that the determination unit 2501, the encoding unit 2502, the first prediction unit 2503 and the second prediction unit 2504 may be software modules or hardware circuits.
[0798] The apparatus for encoding a video in this embodiment can be used to perform Fig.16 The method in Fig.16 The structure of the apparatus for encoding a video involved in the method shown can also be the same as Fig.26 The apparatus for encoding the video shown is the same and will not be described again for the sake of brevity.
[0799] It should be noted that the execution Figures 13 to 16 The method of the device or Figure 23 to Figure 26 The device for encoding a video may be or include Figure 1A Source device 12, Figure 1B Video encoder 20 in Figure 2 Encoder 20 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0800] The present application also provides an encoder, comprising a Figures 13 to 16 The processing circuit of the method provided by any embodiment of the present application is shown.
[0801] It should be noted that the encoder in this embodiment may be or include Figure 1A Source device 12, Figure 1B Video encoder 20 in Figure 2 Encoder 20 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0802] The present application also provides a decoder, comprising a Figures 7 to 12 The processing circuit of the method provided by any embodiment of the present application is shown.
[0803] It should be noted that the decoder in this embodiment may be or include Figure 1A Destination device 14, Figure 1B The video decoder 30 in Figure 3 Decoder 30 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0804] The present application also provides a computer program product including program code. When the program code is executed on a computer or processor, the program code is used to execute Figures 7 to 16 The method provided by any embodiment of the present application is shown.
[0805] The present application also provides a decoder. The decoder includes: one or more processors; a non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program for execution by the one or more processors. When the program is executed by the one or more processors, the program configures the decoder to execute Figures 7 to 12 The method provided by any embodiment of the present application is shown.
[0806] It should be noted that the decoder in this embodiment may be or include Figure 1ADestination device 14, Figure 1B The video decoder 30 in Figure 3 Decoder 30 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0807] The present application also provides an encoder. The encoder includes: one or more processors; a non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program for execution by the one or more processors. When the program is executed by the one or more processors, the program configures the encoder to execute Figures 13 to 16 The method provided by any embodiment of the present application is shown.
[0808] It should be noted that the encoder in this embodiment may be or include Figure 1A Source device 12, Figure 1B Video encoder 20 in Figure 2 Encoder 20 in Figure 4 The video decoding device 400 or Figure 5 In the device 500.
[0809] The present application also provides a non-transitory computer-readable medium including program code. When the program code is executed by a computer device, the program code causes the computer device to execute Figures 7 to 16 The method provided by any embodiment of the present application is shown.
[0810] The present application also provides a non-transient storage medium. The non-transient storage medium includes an encoded code stream decoded by an image decoding device, the code stream includes at least one layer of encoded data, and the code stream also includes a chroma format-related syntax element of a layer indexed by i and a chroma format-related syntax element of a layer indexed by j, wherein, when the layer indexed by j is a reference layer of the layer indexed by i, the value of the chroma format-related syntax element of the layer indexed by i is the same as the value of the chroma format-related syntax element of the layer indexed by j, and i and j are both integers and greater than or equal to 0.
[0811] The present application also provides a non-transitory storage medium. The non-transitory storage medium includes an encoded code stream decoded by an image decoding device, the code stream includes at least one layer of encoded data, and the code stream also includes a bit depth-related syntax element of a layer indexed by i and a bit depth-related syntax element of a layer indexed by j, wherein when the layer indexed by j is a reference layer of the layer indexed by i, the value of the bit depth-related syntax element of the layer indexed by i is the same as the value of the bit depth-related syntax element of the layer indexed by j, and i and j are both integers and greater than or equal to 0.
[0812] In one embodiment, the bit-depth related syntax element represents the bit depth of luma and chroma samples of the image in the layer to which the bit-depth related syntax element applies.
[0813] In one embodiment, the code stream further includes a reference layer syntax element, and the layer with index j is a reference layer of the layer with index i, including: the value of the reference layer syntax element indicates that the layer with index j is a direct reference layer of the layer with index i.
[0814] The following explains the application of the encoding method and the decoding method shown in the above embodiments and the system using these applications.
[0815] Fig. 27 31 is a block diagram of a content supply system 3100 for implementing a content distribution service. The content supply system 3100 includes a capture device 3102, a terminal device 3106, and optionally a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication link 3104. The communication link may include the communication channel 13 described above. The communication link 3104 includes, but is not limited to, Wi-Fi, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any combination thereof.
[0816] Capture device 3102 is used to generate data, and data can be encoded by the coding method shown in the above embodiment. Optionally, capture device 3102 can distribute data to a streaming media server (not shown in the figure), and the server encodes the data and sends the encoded data to terminal device 3106. Capture device 3102 includes but is not limited to a camera, a smart phone or a tablet computer, a computer or a notebook computer, a video conferencing system, a PDA, a vehicle-mounted device or any combination thereof. For example, capture device 3102 may include source device 12 described above. When data includes video, the video encoder 20 included in capture device 3102 can actually perform video encoding. When data includes audio (i.e., sound), the audio encoder included in capture device 3102 can actually perform audio encoding. For some actual scenes, capture device 3102 distributes encoded video data and encoded audio data by multiplexing encoded video data and encoded audio data together. For other actual scenes, for example, in a video conferencing system, encoded audio data and encoded video data are not multiplexed. The capture device 3102 distributes the encoded audio data and the encoded video data to the terminal device 3106 .
[0817] In the content supply system 3100, the terminal device 310 receives and regenerates the encoded data. The terminal device 3106 may be a device having data receiving and recovery capabilities, such as a smart phone or tablet computer 3108, a computer or laptop computer 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a television 3114, a set top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a personal digital assistant (PDA) 3122, a vehicle-mounted device 3124, or a combination thereof, or a device capable of decoding the above-mentioned encoded data. For example, the terminal device 3106 may include the destination device 14 described above. When the encoded data includes video, the video decoder 30 included in the terminal device preferentially performs video decoding. When the encoded data includes audio, the audio decoder included in the terminal device preferentially performs audio decoding processing.
[0818] For a terminal device with a display, such as a smartphone or tablet 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a personal digital assistant (PDA) 3122, or a vehicle-mounted device 3124, the terminal device can feed the decoded data to its display. For a terminal device without a display, such as an STB 3116, a video conferencing system 3118, or a video surveillance system 3120, an external display 3126 is connected therein to receive and display the decoded data.
[0819] When each device in the system performs encoding or decoding, the image encoding device or the image decoding device as shown in the above-mentioned embodiments can be used.
[0820] Fig.2831 is a schematic diagram of an exemplary structure of the terminal device 3106. After the terminal device 3106 receives the stream from the capture device 3102, the protocol processing unit 3202 analyzes the transmission protocol of the stream. The protocol includes but is not limited to Real Time Streaming Protocol (RTSP), Hyper Text Transfer Protocol (HTTP), HTTP Live streaming protocol (HLS), MPEG-DASH, Real-time Transport Protocol (RTP), Real-time Messaging Protocol (RTMP), or any combination thereof, etc.
[0821] After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to the demultiplexing unit 3204. The demultiplexing unit 3204 can divide the multiplexed data into encoded audio data and encoded video data. As mentioned above, for other actual scenes, for example, in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. In this case, the encoded data is not sent to the video decoder 3206 and the audio decoder 3208 by the demultiplexing unit 3204.
[0822] Through the demultiplexing process, a video elementary stream (video ES), an audio ES, and optional subtitles are generated. The video decoder 3206 includes the video decoder 30 described in the above embodiment, and decodes the video ES by the decoding method shown in the above embodiment to generate a video frame, and feeds this data to the synchronization unit 3212. The audio decoder 3208 decodes the audio ES to generate an audio frame, and feeds this data to the synchronization unit 3212. Optionally, the video frame can be stored in a buffer ( Fig.28 Similarly, the audio frames may be stored in a buffer (not shown) before being fed to the synchronization unit 3212. Fig.28 (not shown).
[0823] The synchronization unit 3212 synchronizes the video frames and the audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 3212 synchronizes the presentation of the video information and the audio information. The information may be coded in syntax using timestamps associated with the presentation of the decoded audio and visual data and timestamps associated with the transmission of the data stream itself.
[0824] If subtitles are included in the stream, the subtitle decoder 3210 decodes the subtitles, synchronizes the subtitles with the video frames and the audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 3216.
[0825] The present invention is not limited to the above-mentioned system, and the image encoding device or the image decoding device in the above-mentioned embodiments may be included in other systems such as automobile systems.
[0826] It should be noted that the content supply system 3100 and the terminal device 3106 are used to execute the above Figures 7 to 16 The method in .
[0827] Mathematical operators
[0828] The mathematical operators used in this application are similar to those used in the C programming language. However, the present invention accurately defines the results of integer division and arithmetic shift operations, and also defines other operations, such as power operations and real-valued division. Numbering and counting specifications usually start from 0, for example, "first" is equivalent to the 0th, "second" is equivalent to the 1st, and so on.
[0829] Arithmetic operators
[0830] The arithmetic operators are defined as follows:
[0831] + Addition
[0832] – Subtraction (binary operator) or negation (unary prefix operator)
[0833] * Multiplication operations, including matrix multiplication
[0834]
[0835] ÷ Division operation without truncation or rounding.
[0836] Division operation without truncation or rounding.
[0837] The cumulative sum of the function f(i) for all integer values of i from x to y (inclusive).
[0838] x%y Modulo operation, which means the remainder when x is divided by y, where x and y are both integers, and x>=0, y>0.
[0839] Logical operators
[0840] The logical operators are defined as follows:
[0841] x&&y Boolean logic "AND" operation of x and y
[0842] x||y Boolean logical "OR" operation of x and y
[0843] ! Boolean logic "not" operation
[0844] x? y:z If x is true or not equal to 0, find the value of y; otherwise, find the value of z.
[0845] Relational operators
[0846] The relational operators are defined as follows:
[0847] > Greater than
[0848] >= greater than or equal to
[0849] < less than
[0850] <= less than or equal to
[0851] == is equal to
[0852] ! = Not equal to
[0853] When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is considered a different value for the syntax element or variable. The value "na" is considered not equal to any other value.
[0854] Bitwise Operators
[0855] The bitwise operators are defined as follows:
[0856] & Operates bitwise AND. When operating on integer arguments, the operation is performed on the two's complement representation of the integer value. When operating on binary arguments, if it consists of fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.
[0857] | Operates bitwise OR. When operating on integer arguments, the operation is performed on the two's complement representation of the integer value. When operating on binary arguments, if it consists of fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to 0.
[0858] ^ Bitwise exclusive-or. When operating on integer arguments, the two's complement representation of the integer value is operated on. When operating on binary arguments, if it consists of fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to zero.
[0859] x >> y shifts x arithmetically right by y bit positions in its two's complement integer representation. This function is only defined when y is a non-negative integer value. The result of the right shift is that the bit shifted into the most significant bit (MSB) equals the MSB of x before the shift operation.
[0860] x << y shifts x arithmetically left by y bit positions in its two's complement integer representation. This function is only defined when y is a non-negative integer value. The result of the left shift is that the bit shifted into the least significant bit (LSB) equals 0.
[0861] Assignment operators
[0862] The arithmetic operators are defined as follows:
[0863] = Assignment operator
[0864] ++ Increment, i.e., x++ is equivalent to x = x + 1; when used as an array subscript, the value of the variable is evaluated before the increment operation.
[0865] –– Decrement, i.e., x–– is equal to x = x – 1; when used as an array subscript, the value of the variable is evaluated before the decrement operation.
[0866] += Add a specified value, i.e., x += 3 is equivalent to x = x + 3, and x += (–3) is equivalent to x = x + (–3).
[0867] –= Subtract a specified value, i.e., x –= 3 is equivalent to x = x – 3, and x –= (–3) is equivalent to x = x – (–3).
[0868] Range notation
[0869] The following notation is used to indicate the range of values:
[0870] x = y..z x takes integer values from y to z (including the end values), where x, y, and z are all integers and z is greater than y.
[0871] Mathematical functions
[0872] The mathematical functions are defined as follows:
[0873]
[0874] Asin(x) The arc sine function, which operates on the argument x, where x is in the range –1.0 to 1.0 (including the end values), and the output value is in the range –π÷2 to π÷2 (including the end values), in radians.
[0875] Atan(x) Trigonometric inverse tangent function. It operates on the parameter x and the output value is in the range of –π÷2 to π÷2 (including the end values) and the unit is radians.
[0876]
[0877] Ceil(x) The smallest integer greater than or equal to x.
[0878] Clip1 Y (x)=Clip3(0,(1< <BitDepth Y )–1,x)
[0879] Clip1 C (x)=Clip3(0,(1< <BitDepth C )–1,x)
[0880]
[0881] Cos(x) Trigonometric cosine function, operates on parameter x, in radians.
[0882] Floor(x) The largest integer less than or equal to x.
[0883]
[0884] Ln(x) The natural logarithm of x (logarithm with base e, where e is the base constant of natural logarithms 2.718281828…).
[0885] Log2(x) is the base 2 logarithm of x.
[0886] Log10(x) is the base 10 logarithm of x.
[0887]
[0888]
[0889] Round(x)=Sign(x)*Floor(Abs(x)+0.5)
[0890]
[0891] Sin(x) Trigonometric sine function, operates on parameter x, in radians.
[0892]
[0893] Swap(x,y)=(y,x)
[0894] Tan(x) is the trigonometric tangent function, which operates on the parameter x in radians.
[0895] Operation priority order
[0896] When parentheses are not used to explicitly indicate the order of precedence in an expression, the following rules apply:
[0897] – Operations with higher priority are evaluated before any operations with lower priority.
[0898] – Operations of the same priority are evaluated from left to right.
[0899] The following table describes the priority of operations from highest to lowest. The higher the position in the table, the higher the priority.
[0900] For operators that are also used in the C programming language, the operator precedence order in this specification is the same as the precedence order in the C programming language.
[0901] Table: Operation priority is sorted from highest (top of table) to lowest (bottom of table)
[0902]
[0903] Text description of the logical operation
[0904] In the text, the following logical operation statements are described in mathematical form:
[0905] if(condition 0)
[0906] statement 0
[0907] else if(condition 1)
[0908] statement 1 ...
[0910] else / *informative remark on remaining condition* /
[0911] statement
[0912] It can be described in the following way:
[0913] ...as follows / ...the following applies:
[0914] – If condition 0 then statement 0
[0915] – Else if condition 1 then statement 1
[0916] …
[0917] – Otherwise (instructions for the remaining conditions), statement n
[0918] Each "if ... else, if ... else, ..." statement in the text is introduced with "... as follows" or "... the following applies" followed by "if ...". The last condition of "if ... else, if ... else, ..." always has an "otherwise, ...". Statements with "if ... else, if ... else" in the middle can be identified by matching "... as follows" or "... the following applies" with the ending "otherwise ...".
[0919] In the text, the following logical operation statements are described in mathematical form:
[0920] if(condition 0a&&condition 0b)
[0921] statement 0
[0922] else if(condition 1a||condition 1b)
[0923] statement 1 ...
[0925] else
[0926] statement
[0927] It can be described in the following way:
[0928] ...as follows / ...the following applies:
[0929] – Statement 0 if all of the following are true:
[0930] –Condition 0a
[0931] – Condition 0b
[0932] – If one or more of the following conditions are met, then statement 1:
[0933] –Condition 1a
[0934] – Condition 1b
[0935] -……
[0936] – Otherwise, statement n
[0937] In the text, the following logical operation statements are described in mathematical form:
[0938] if(condition 0)
[0939] statement 0
[0940] if(condition 1)
[0941] statement 1
[0942] It can be described in the following way:
[0943] When condition 0, statement 0
[0944] When condition 1, statement 1.
[0945] Although the embodiments of the present invention are mainly described in terms of video decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 (respectively, the system 10) and other embodiments described herein can also be used for still image processing or decoding, that is, processing or decoding a single image in video decoding that is independent of any previous or consecutive images. In general, if the image processing decoding is limited to a single image 17, only the inter-frame prediction unit 244 (encoder) and 344 (decoder) are not available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270 and entropy decoding 304.
[0946] Embodiments of encoders 20, decoders 30, etc. and the functions described herein with reference to encoders 20, decoders 30, etc. may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or codes in a computer-readable medium or sent via a communication medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media (e.g., data storage media), or any communication media that facilitates sending a computer program from one place to another according to a communication protocol, etc. In this manner, computer-readable media may generally correspond to (1) non-transitory tangible computer-readable storage media or (2) communication media such as signals or carrier waves. Data storage media may be any available media accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in the present invention. A computer program product may include a computer-readable medium.
[0947] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, flash memory or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by the computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instructions are transmitted from a website, server or other remote source via a coaxial cable, optical fiber cable, twisted pair and digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, optical fiber cable, twisted pair and DSL, or wireless technologies such as infrared, radio and microwave are also included in the definition of the above medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other transient media, but rather involve non-transient tangible storage media. Disks and optical disks used herein include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs) and blue-ray discs, wherein disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0948] Instructions can be executed by one or more processors such as one or more digital signal processors (DSP), one or more general-purpose microprocessors, one or more application specific integrated circuits (ASIC), one or more field programmable logic arrays (FPGA) or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" used herein may refer to any of the above structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the various functions described herein may be provided in dedicated hardware and / or software modules for encoding and decoding, or incorporated into a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.
[0949] The techniques of the present invention may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., chipsets). The various components, modules, or units described in the present invention are intended to emphasize the functional aspects of the apparatus for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by a collection of interoperable hardware units including one or more processors as described above.
Claims
1. A method for encoding a video, characterized in that The method comprises: Determine whether a layer with index j is a reference layer for a layer with index i, where i and j are both integers greater than or equal to 0; If the layer with index j is the reference layer of the layer with index i, a reference layer syntax element whose value indicates that the layer with index j is the reference layer of the layer with index i is encoded into the video bitstream, and a chroma format related syntax element applied to the layer with index i and a chroma format related syntax element applied to the layer with index j are encoded into the video bitstream, and a bit depth related syntax element applied to the layer with index i and a bit depth related syntax element applied to the layer with index j are encoded into the video bitstream, wherein the chroma format applied to the layer with index i is The value of the relevant syntax element is the same as the value of the chroma format-related syntax element applied to the layer with index j, and the value of the bit depth-related syntax element applied to the layer with index i and the value of the bit depth-related syntax element applied to the layer with index j are the same, wherein the same value of the bit depth-related syntax element applied to the layer with index i and the bit depth-related syntax element applied to the layer with index j indicates that inter-layer prediction can only be used in the scenario where the value of the bit depth-related syntax element of the layer with index i and the value of the bit depth-related syntax element applied to the layer with index j are the same.
2. The method according to claim 1, characterized in that The method further comprises: If the layer with index j is a reference layer of the layer with index i, the image in the layer with index i is predicted based on the layer with index j.
3. The method according to claim 1, characterized in that The method further comprises: If the layer with index j is not a reference layer of the layer with index i, the layer with index j is not used to predict the image in the layer with index i.
4. A device for encoding a video, characterized in that The device comprises: a determining unit, configured to determine whether a layer with an index of j is a reference layer of a layer with an index of i, wherein i and j are both integers and are greater than or equal to 0; A coding unit, configured to: if the layer with index j is the reference layer of the layer with index i, encode into a video bitstream a reference layer syntax element whose value indicates that the layer with index j is the reference layer of the layer with index i, and encode into the video bitstream a chroma format-related syntax element applied to the layer with index i and a chroma format-related syntax element applied to the layer with index j, and encode into the video bitstream a bit depth-related syntax element applied to the layer with index i and a bit depth-related syntax element applied to the layer with index j, wherein the syntax element applied to the layer with index i is a bit depth-related syntax element applied to the layer with index j. The value of the chroma format-related syntax element is the same as the value of the chroma format-related syntax element applied to the layer with index j, and the value of the bit depth-related syntax element applied to the layer with index i is the same as the value of the bit depth-related syntax element applied to the layer with index j, wherein the same value of the bit depth-related syntax element applied to the layer with index i and the bit depth-related syntax element applied to the layer with index j indicates that inter-layer prediction can only be used in the scenario where the value of the bit depth-related syntax element of the layer with index i and the value of the bit depth-related syntax element applied to the layer with index j are the same.
5. The device according to claim 4, characterized in that The device also includes a first prediction unit; The first prediction unit is configured to: if the layer with index j is a reference layer of the layer with index i, predict the image in the layer with index i according to the layer with index j.
6. The device according to claim 4, characterized in that The device also includes a second prediction unit; The second prediction unit is configured to: if the layer with index j is not a reference layer of the layer with index i, not use the layer with index j to predict the image in the layer with index i.
7. An encoder, characterized in that The encoder comprises processing circuitry for performing the method according to any one of claims 1 to 3.
8. An encoder, wherein: The encoder comprises: one or more processors; A non-transitory computer-readable storage medium, coupled to the one or more processors and storing a program executed by the one or more processors, wherein when the program is executed by the one or more processors, the program configures the encoder to perform the method according to any one of claims 1 to 3.
9. A non-transitory computer-readable medium comprising a program code, characterized in that: When the program code is executed by a computer device, the program code causes the computer device to perform the method according to any one of claims 1 to 3.
