enhancement signaling notification of supplemental enhancement information

Abstract

A mechanism for processing video data is disclosed. A value of an annotation region (AR) object label index of an i-th AR object index (ar_object_label_idx[ar_object_idx[i]]) syntax element is determined. The ar_object_label_idx[ar_object_idx[i]] syntax element is specified in a range of N to M, where N and M are integers and N is less than M. A conversion between a visual media data and a bitstream is performed based on the ar_object_label_idx[ar_object_idx[i]] syntax element.

Description

[0001] Cross-reference to related applications

[0002] This application is the national phase application of International Patent Application No. PCT / CN2022 / 101411, filed on June 27, 2022, which claims the benefit of International Patent Application No. PCT / CN2021 / 102636, filed on June 28, 2021. All of the above-mentioned patent applications are incorporated herein by reference in their entirety. Technical Field

[0003] This patent document relates to the generation, storage, and consumption of digital audio and video media information in file formats. Background Technology

[0004] Digital video accounts for the largest share of bandwidth usage on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video is expected to continue to grow. Summary of the Invention

[0005] The first aspect relates to a method for processing video data, comprising: determining that the value of the annotation region (AR) object label index of the i-th AR object index (ar_object_label_idx[ar_object_idx[i]]) syntax element is specified in the range N to M, where N and M are integers and N is less than M; and performing a conversion between visual media data and a bitstream based on the ar_object_label_idx[ar_object_idx[i]] syntax element.

[0006] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides N as 0 and M as 3, 7, 15, 31, 63, or 255.

[0007] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides the range N to M including end values.

[0008] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that the value of the depth_representation_type syntax element is specified to be within the range of A to B, where A and B are integers and A is less than B.

[0009] Optionally, in any of the foregoing aspects, another implementation of that aspect provides A as 0 and B as 3, 7, 15, 31, 63, 127, or 255.

[0010] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides the range from A to B to include end values.

[0011] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that the value of the syntactic element of the i-th depth nonlinear representation model (depth_nonlinear_representation_model[i]) is specified in the range C to D, where C and D are integers and C is less than D.

[0012] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides C as 0 and D as 6, 14, 30, 62, 126, or 254.

[0013] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides that the range from C to D includes end values.

[0014] Alternatively, in any of the foregoing aspects, another implementation of the aspect provides that the value of the disparity reference view identifier (disparity_ref_view_id) syntax element is specified in the range E to F, where E and F are integers and E is less than F.

[0015] Optionally, in any of the foregoing aspects, another implementation of the aspect provides E as 0 and F as 63, 127, 255, 511, 1023, 2047, 4095, 8191, 16383, 32767 or 65535.

[0016] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides the range from E to F to include end values.

[0017] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that the ar_object_label_idx[ar_object_idx[i]] syntax element is included in the Annotated Region (AR) Supplemental Enhancement Information (SEI) message.

[0018] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides that the depth_representation_type syntax element is included in the Depth Representation Information (DRI) SEI message.

[0019] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides that the depth_nonlinear_representation_model[i] syntax element is included in the DRI SEI message.

[0020] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides that the disparity_ref_view_id syntax element is included in the DRI SEI message.

[0021] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides a conversion that includes encoding visual media data into a bitstream.

[0022] Alternatively, in any of the foregoing aspects, another implementation of that aspect provides a conversion that includes decoding the bitstream to obtain visual media data.

[0023] The second aspect relates to an apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method described in any of the preceding aspects.

[0024] The third aspect relates to a non-transitory computer-readable medium comprising a computer program product for use by a video codec apparatus, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, such that when executed by a processor, the computer-executable instructions cause the video codec apparatus to perform the method described in any of the preceding aspects.

[0025] The fourth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of video, the bitstream of which is generated by a method performed by a video processing apparatus, the method comprising: determining the value of the annotation region (AR) object label index of the i-th AR object index (ar_object_label_idx[ar_object_idx[i]]) syntax element, the ar_object_label_idx[ar_object_idx[i]] syntax element being specified in the range N to M, wherein N and M are integers and N is less than M; and generating the bitstream based on the determination.

[0026] The fifth aspect relates to a method for storing a bitstream of video, comprising: determining the value of the annotation region (AR) object label index of the i-th AR object index (ar_object_label_idx[ar_object_idx[i]]) syntax element, wherein the ar_object_label_idx[ar_object_idx[i]] syntax element is specified in the range N to M, where N and M are integers and N is less than M; generating a bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0027] For clarity, any of the foregoing embodiments may be combined with any one or more other foregoing embodiments to form new embodiments within the scope of this disclosure.

[0028] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0029] To gain a more complete understanding of the contents of this disclosure, the following brief description is provided for reference in conjunction with the accompanying drawings and detailed specifications, wherein the same reference numerals represent the same parts.

[0030] Figure 1 This is a schematic diagram of an example bitstream.

[0031] Figure 2 This is a block diagram illustrating an example video processing system.

[0032] Figure 3 This is a block diagram of an example video processing device.

[0033] Figure 4 This is a flowchart of an example method for video processing.

[0034] Figure 5 This is a block diagram illustrating an example video encoding / decoding system.

[0035] Figure 6 This is a block diagram of an example encoder.

[0036] Figure 7 This is a block diagram of an example decoder.

[0037] Figure 8 This is a schematic diagram of an example encoder. Detailed Implementation

[0038] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or yet to be developed. This disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but modifications can be made within the full scope of the appended claims and their equivalents.

[0039] This patent document relates to image and / or video codec technologies. Specifically, this document relates to signaling notifications of annotated regions, depth representation information, and Extended Dependent Random Access Point (EDRAP) indications in Supplemental Enhancement Information (SEI) messages. These examples can be applied individually or in various combinations for video bitstreams encoded and decoded by any codec, such as the Multi-Functional Video Codec (VVC) standard and the Multi-Functional SEI Message (VSEI) standard for encoding and decoding video bitstreams.

[0040] This disclosure includes the following abbreviations: Alpha Channel Information (ACI), Adaptive Parameter Set (APS), Access Unit (AU), Codec Layer Video Sequence (CLVS), Codec Layer Video Sequence Start (CLVSS), Cyclic Redundancy Check (CRC), Color Transformation Information (CTI), Codec Video Sequence (CVS), Dependent Random Access Point (DRAP), Depth Representation Information (DRI), Extended Dependent Random Access Point (EDRAP), Finite Impulse Response (FIR), Intra-Frame Random Access Point (IRAP), Multi-View Acquisition Information (MAI), Network Abstraction Layer (NAL), Picture Parameter Set (PPS), Picture Unit (PU), Random Access Skip Preamble (RASL), Region Packing (RWP), Sample Aspect Ratio (SAR), Sample Aspect Ratio Information (SARI), Scalability Dimension Information (SDI), Supplemental Enhancement Information (SEI), Sequential Temporal Sublayer Access (STSA), Video Codec Layer (VCL), Multifunctional Supplemental Enhancement Information, also known as Rec.ITU-T H.274|ISO / IEC 23002-7 (VSEI), Video Availability Information (VUI), and Multi-Functional Video Codec, also known as Rec.ITU-T H.266|ISO / IEC 23090-3 (VVC).

[0041] Video codec standards have primarily evolved through the development of standards by the International Telecommunication Union (ITU) Telecommunication Standardization Sector (ITU-T) and the ISO / IEC. ITU-T developed H.261 and H.263, while ISO / IEC developed Moving Picture Experts Group (MPEG)-1 and MPEG-4 video. These two organizations jointly developed the H.262 / MPEG-2 video standard and the H.264 / MPEG-4 Advanced Video Codec (AVC) and H.265 / High-Efficiency Video Codec (HEVC) standards. Since H.262, video codec standards have been based on a hybrid video codec architecture, utilizing temporal prediction plus transform coding. To further explore video codec technologies beyond HEVC, the Video Codec Experts Group (VCEG) and MPEG jointly established the Joint Video Exploration Team (JVET). JVET adopted numerous methodologies and incorporated them into reference software called the Joint Exploration Model (JEM). When the Multifunctional Video Codec (VVC) project was officially launched, JVET was later renamed the Joint Video Experts Team (JVET). VVC is a codec standard designed to reduce the bit rate by 50% compared to HEVC. VVC has been finalized by JVET.

[0042] The VVC standard, also known as ITU-T H.266|ISO / IEC 23090-3, and its related Multifunctional Supplemental Enhancement Information (VSEI) standard, also known as ITU-T H.274|ISO / IEC 23002-7, are designed for a wide range of applications, such as television broadcasting, video conferencing, storage media playback, adaptive bitrate streaming, video region extraction, merging and combining content from multiple codec video bitstreams, multi-view video, scalable layered codecs, and viewport-adaptive 360° immersive media. The Basic Video Codec (EVC) standard (ISO / IEC 23094-1) is another video codec standard developed by MPEG.

[0043] Examples of modifications to the VSEI standard include the specification of additional SEI messages, which include annotation area SEI messages, alpha channel information SEI messages, depth representation information SEI messages, multi-view acquisition information SEI messages, scalability dimension information SEI messages, extended dependent random access point (DRAP) indication SEI messages, display orientation SEI messages, and color transformation information SEI messages.

[0044] The following is an example of the SEI message syntax for the comment area.

[0045]

[0046]

[0047]

[0048] The following is an example of the semantics of the Annotated Region SEI message. The Annotated Region SEI message carries parameters that identify the annotated region using bounding boxes representing the size and position of the identified object. Using this SEI message may require defining the following variables: These variables include the image width and height cropped in units of luminance samples, denoted here as CroppedWidth and CroppedHeight, respectively; the chroma subsample width and height, denoted as SubWidthC and SubHeightC, respectively; the left offset of the consistent cropping window, denoted as ConfWinLeftOffset; and the top offset of the consistent cropping window, denoted as ConfWinTopOffset.

[0049] Setting `ar_cancel_flag` to 1 indicates that the annotated region SEI message cancels the persistence of any previous annotated region SEI messages associated with one or more layers to which the annotated region SEI message was applied. Setting `ar_cancel_flag` to 0 indicates that annotated region information follows. When `ar_cancel_flag` equals 1 or a new CVS for the current layer begins, the variables `LabelAssigned[i]`, `ObjectTracked[i]`, and `ObjectBoundingBoxAvail` are set to 0 for `i` in the range 0 to 255 (inclusive).

[0050] Setting `ar_not_optimized_for_viewing_flag` to 1 indicates that the decoded image used for the SEI message in the annotated region is not optimized for user viewing, but rather optimized for other purposes related to the performance of the algorithm's object classification. Setting `ar_not_optimized_for_viewing_flag` to 0 indicates that the decoded image used for the SEI message in the annotated region may or may not be optimized for user viewing.

[0051] Setting `ar_true_motion_flag` to 1 indicates that the motion information in the codec image used by the SEI message for the annotation region is selected with the goal of accurately representing the object motion of the objects in the annotation region. Setting `ar_true_motion_flag` to 0 indicates that the motion information in the codec image used by the SEI message for the annotation region can be selected with the goal of accurately representing the object motion of the objects in the annotation region, or it can be selected without the goal of accurately representing the object motion of the objects in the annotation region.

[0052] Setting `ar_occluded_object_flag` to 1 indicates that each of the `ar_bounding_box_top[ar_object_idx[i]]`, `ar_bounding_box_left[ar_object_idx[i]]`, `ar_bounding_box_width[ar_object_idx[i]]`, and `ar_bounding_box_height[ar_object_idx[i]]` syntax elements represents the size and position of an object or a portion of an object that may be invisible or only partially visible within the cropped decoded image. Setting `ar_occluded_object_flag` to 0 indicates that the `ar_bounding_box_top[ar_object_idx[i]]`, `ar_bounding_box_left[ar_object_idx[i]]`, `ar_bounding_box_width[ar_object_idx[i]]`, and `ar_bounding_box_height[ar_object_idx[i]]` syntax elements represent the size and position of an object that is fully visible within the cropped decoded image. Bitstream consistency requires that the value of ar_occluded_object_flag be the same for all annotated_regions() syntax structures within CVS.

[0053] Setting `ar_partial_object_flag_present_flag` to 1 indicates that the syntax element `ar_partial_object_flag[ar_object_idx[i]]` exists. Setting `ar_partial_object_flag_present_flag` to 0 indicates that the syntax element `ar_partial_object_flag[ar_object_idx[i]]` does not exist. Bitstream consistency requires that the value of `ar_partial_object_flag_present_flag` should be the same for all `annotated_regions()` syntax structures within CVS.

[0054] Setting `ar_object_label_present_flag` to 1 indicates that there is label information corresponding to the object in the comment area. Setting `ar_object_label_present_flag` to 0 indicates that there is no label information corresponding to the object in the comment area.

[0055] Setting `ar_object_confidence_info_present_flag` to 1 indicates the existence of the `ar_object_confidence[ar_object_idx[i]]` syntax element. Setting `ar_object_confidence_info_present_flag` to 0 indicates the absence of the `ar_object_confidence[ar_object_idx[i]]` syntax element. Bitstream consistency requires that the value of `ar_object_confidence_present_flag` should be identical for all `annotated_regions()` syntax structures within CVS.

[0056] `ar_object_confidence_length_minus 1+1` indicates the length (in bits) of the `ar_object_confidence[ar_object_idx[i]]` syntax element. Bitstream consistency requires that the value of `ar_object_confidence_length_minus 1` should be the same for all `annotated_regions()` syntax structures within CVS.

[0057] Setting `ar_object_label_language_present_flag` to 1 indicates the existence of the `ar_object_label_language` syntax element. Setting `ar_object_label_language_present_flag` to 0 indicates the absence of the `ar_object_label_language` syntax element. `ar_bit_equal_to_zero` should be equal to zero.

[0058] The `ar_object_label_language` parameter contains the language label, followed by a null terminator byte equal to 0x00. The length of the `ar_object_label_language` syntax element should be less than or equal to 255 bytes, excluding the null terminator byte. If it is not present, the language of the label is unspecified.

[0059] `ar_num_label_updates` indicates the total number of labels associated with the annotation area to be signaled. The value of `ar_num_label_updates` should be in the range of 0 to 255 (inclusive). `ar_label_idx[i]` indicates the index of the label to be signaled. The value of `ar_label_idx[i]` should be in the range of 0 to 255 (inclusive).

[0060] Setting `ar_label_cancel_flag` to 1 cancels the persistent range of the `ar_label_idx[i]` label. Setting `ar_label_cancel_flag` to 0 indicates that the `ar_label_idx[i]` label has been assigned a signaling notification value. `ar_label[ar_label_idx[i]]` specifies the content of the `ar_label_idx[i]` label. The length of the `ar_label[ar_label_idx[i]]` syntax element should be less than or equal to 255 bytes, excluding the null terminator.

[0061] `ar_num_object_updates` indicates the number of object updates to be signaled. `ar_num_object_updates` should be in the range of 0 to 255 (inclusive). `ar_object_idx[i]` is the index of the object parameter to be signaled. `ar_object_idx[i]` should be in the range of 0 to 255 (inclusive). Setting `ar_object_cancel_flag` to 1 cancels the persistence range of the `ar_object_idx[i]`-th object. Setting `ar_object_cancel_flag` to 0 indicates that parameters associated with the object tracked by the `ar_object_idx[i]`-th object should be signaled. Setting `ar_object_label_update_flag` to 1 indicates that the object label is signaled. Setting `ar_object_label_update_flag` to 0 indicates that the object label is not signaled.

[0062] `ar_object_label_idx[ar_object_idx[i]]` indicates the index of the label corresponding to the `ar_object_idx[i]`-th object. When `ar_object_label_idx[ar_object_idx[i]]` does not exist, its value is inferred from the previous annotated region SEI messages in the same CVS (if any) in output order. `ar_bounding_box_update_flag` set to 1 indicates that the object bounding box parameter is signaled. `ar_bounding_box_update_flag` set to 0 indicates that the object bounding box parameter is not signaled.

[0063] Set ar_bounding_box_cancel_flag equal to 1 to cancel ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_ob The persistence scope of ject_idx[i]], ar_bounding_box_height[ar_object_idx[i]], ar_partial_object_flag[ar_object_idx[i]], and ar_object_confidence[ar_object_idx[i]]. Setting ar_bounding_box_cancel_flag to 0 indicates that the syntax elements ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], ar_bounding_box_height[ar_object_idx[i]], ar_partial_object_flag[ar_object_idx[i]], and ar_object_confidence[ar_object_idx[i]] are signaled.

[0064] ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] specify the left top corner coordinates, width, and height of the bounding box of the ar_object_idx[i]-th object in the decoded image relative to the consistent clipping window defined by the active SPS.

[0065] The value of `ar_bounding_box_left[ar_object_idx[i]]` should be in the range of 0 to `CroppedWidth / SubWidthC–1` (inclusive). The value of `ar_bounding_box_top[ar_object_idx[i]]` should be in the range of 0 to `CroppedHeight / SubHeightC–1` ​​(inclusive). The value of `ar_bounding_box_width[ar_object_idx[i]]` should be in the range of 0 to `CroppedWidth / SubWidthC–ar_bounding_box_left[ar_object_idx[i]]` (inclusive). The value of `ar_bounding_box_height[ar_object_idx[i]]` should be in the range of 0 to `CroppedHeight / SubHeightC–ar_bounding_box_top[ar_object_idx[i]]` (inclusive). The identified object rectangle contains brightness samples with horizontal image coordinates from SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]]) to SubWidthC*(ConfWinLeftOffset+ar_bounding_box_left[ar_object_idx[i]]+ar_bounding_box_width[ar_object_idx[i]])-1 (inclusive of end values) and vertical image coordinates from SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]) to SubHeightC*(ConfWinTopOffset+ar_bounding_box_top[ar_object_idx[i]]+ar_bounding_box_height[ar_object_idx[i]])-1 (inclusive of end values). For each value of ar_object_idx[i], the values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] are retained in CVS in the order of their output.When not present, the values ​​of ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], or ar_bounding_box_height[ar_object_idx[i]] are inferred from the SEI messages of the previously annotated regions in CVS, arranged in output order (if any).

[0066] Setting ar_partial_object_flag[ar_object_idx[i]] to 1 indicates that the syntax elements ar_bounding_box_top[ar_object_idx[i]], ar_bounding_box_left[ar_object_idx[i]], ar_bounding_box_width[ar_object_idx[i]], and ar_bounding_box_height[ar_object_idx[i]] represent the size and position of objects that are only partially visible in the cropped decoded image. Setting `ar_partial_object_flag[ar_object_idx[i]]` to 0 indicates that the syntax elements `ar_bounding_box_top[ar_object_idx[i]]`, `ar_bounding_box_left[ar_object_idx[i]]`, `ar_bounding_box_width[ar_object_idx[i]]`, and `ar_bounding_box_height[ar_object_idx[i]]` represent the size and position of an object that may or may not be partially visible in the cropped decoded image. When not present, the value of `ar_partial_object_flag[ar_object_idx[i]]` is inferred from the previously annotated SEI messages in CVS, arranged in output order (if any).

[0067] `ar_object_confidence[ar_object_idx[i]]` indicates the confidence level associated with the `ar_object_idx[i]`-th object, in units of 2 - (ar_object_confidence_length_minus1 + 1). Therefore, a higher value of `ar_object_confidence[ar_object_idx[i]]` indicates a higher confidence level. The length of the `ar_object_confidence[ar_object_idx[i]]` syntax element is `ar_object_confidence_length_minus1 + 1` bits. If it does not exist, the value of `ar_object_confidence[ar_object_idx[i]]` is inferred from the previously annotated SEI messages in CVS, arranged in output order (if any).

[0068] We will now discuss the Depth Representation Information (SEI) message. An example of the SEI message syntax is shown below.

[0069]

[0070]

[0071] The following is an example of the syntax for representing information elements with depth.

[0072]

[0073] The following is an example of the semantics of the Depth Representation Information (SEI) message. Syntax elements in the Depth Representation Information (DRI) SEI message specify various parameters for an auxiliary picture of type AUX_DEPTH so that the primary and auxiliary pictures are processed and decoded before being rendered on a 3D display, such as in view composition. For example, the depth or parallax range of the depth picture is specified.

[0074] Using this SEI message may require defining the following variables: The bit depth of the sample points for the luminance component, denoted here by BitDepthY. For at least one value of i, no image in the CVS should be associated with a DRI SEI message when the CVS does not contain an SDI SEI message with sdi_aux_id[i] equal to 2. When an access unit (AU) contains both an SDI SEI message with sdi_aux_id[i] equal to 2 (for at least one value of i) and a DRI SEI message, the decoding order of the SDI SEI message should precede the decoding order of the DRI SEI message. When present, a DRI SEI message should be associated with one or more layers indicated as depth auxiliary layers by the SDI SEI message. The following semantics apply to each nuh_layer_id targetLayerId in the nuh_layer_id value to which the DRI SEI message is applied. When present, a DRI SEI message can be included in any access unit. It is recommended that, when an SEI message is present, it be included in an access unit for random access purposes, where the encoded / decoded image with a nuh_layer_id equal to targetLayerId is an IRAP image. The information indicated in the DRI SEI message applies to all images with nuh_layer_id equal to targetLayerId, starting from the access unit containing the SEI message, in the decoding order, until the next image associated with the DRI SEI message applicable to targetLayerId (not included), or until the end of the CLVS with nuh_layer_id equal to targetLayerId, whichever is earlier in the decoding order.

[0075] Setting `z_near_flag` to 0 indicates that the syntax structure does not contain a syntax element specifying the nearest depth value. Setting `z_near_flag` to 1 indicates that the syntax structure contains a syntax element specifying the nearest depth value. Setting `z_far_flag` to 0 indicates that the syntax structure does not contain a syntax element specifying the farthest depth value. Setting `z_far_flag` to 1 indicates that the syntax structure contains a syntax element specifying the farthest depth value. Setting `d_min_flag` to 0 indicates that the syntax structure does not contain a syntax element specifying the minimum disparity value. Setting `d_min_flag` to 1 indicates that the syntax structure contains a syntax element specifying the minimum disparity value. Setting `d_max_flag` to 0 indicates that the syntax structure does not contain a syntax element specifying the maximum disparity value. Setting `d_max_flag` to 1 indicates that the syntax structure contains a syntax element specifying the maximum disparity value. `depth_representation_type` specifies the representation definition of the decoded luminance samples of the auxiliary image as specified in Table 1. In Table 1, disparity specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera. The variable `maxVal` is set to equal to (1 < ... <BitDepthY)–1。

[0076]

[0077]

[0078] Table 1 – Definition of depth_representation_type

[0079] `disparity_ref_view_id` specifies the ViewId value from which the disparity value is derived. It should be noted that `disparity_ref_view_id` only exists when `d_min_flag` equals 1 or `d_max_flag` equals 1, and is useful for `depth_representation_type` values ​​of 1 and 3. The variables in column x of Table 2 are derived from the corresponding variables in columns s, e, n, and v of Table 2, as follows. If the value of e is in the range of 0 to 127 (excluding the extreme values), then x is set to equal to (-1)s*2e-31*(1+n÷2v). Otherwise (e equals 0), x is set to equal to (-1)s*2-(30+v)*n.

[0080] ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen

[0081] Table 2 – Relationship between depth parameter variables and syntax elements

[0082] When present, the Dmin and DMax values ​​are specified in units of the luminance sample width of the encoded / decoded image, where ViewId is equal to the ViewId of the auxiliary image. When present, the ZNear and ZFar values ​​have the same units but are not specified. depth_nonlinear_representation_num_minus1+2 specifies the number of piecewise linear segments used to map depth values ​​to a scale uniformly quantized according to disparity. depth_nonlinear_representation_model[i] for i ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive) specifies the piecewise linear segments used to map the decoded luminance sample values ​​of the auxiliary image to a scale uniformly quantized according to disparity. The values ​​of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0.

[0083] When `depth_representation_type` equals 3, the auxiliary image contains depth samples with a non-linear transformation. The variable `DepthLUT[i]`, as specified below, is used to transform the decoded depth sample values ​​from a non-linear representation to a linear representation, i.e., uniformly quantized disparity values. The shape of this transformation is defined by approximating a line segment in the non-linear disparity space from a two-dimensional linear disparity. The first (0, 0) and last (maxVal, maxVal) nodes of the curve are predefined. The positions of the auxiliary nodes are sent as deviations from the straight curve (`depth_nonlinear_representation_model[i]`). These deviations are uniformly distributed across the entire range from 0 to maxVal (inclusive), with spacing depending on the value of `nonlinear_depth_representation_num_minus1`.

[0084] The variable DepthLUT[i], whose i is in the range of 0 to maxVal (inclusive), is defined as follows:

[0085]

[0086] When depth_representation_type equals 3, DepthLUT[dS] represents the disparity uniformly quantized to the range of 0 to maxVal (inclusive) for all decoded luminance sample values ​​dS of the auxiliary image in the range of 0 to maxVal (inclusive).

[0087] The semantics of the depth representation information elements are as follows. The syntax structure specifies the values ​​of elements in the DRI SEI message. The syntax structure sets the values ​​of the variables OutSign, OutExp, OutMantissa, and OutManLen, which represent floating-point values. When this syntax structure is contained within another syntax structure, the variable names OutSign, OutExp, OutMantissa, and OutManLen will be interpreted as being replaced by the variable names used when containing this syntax structure.

[0088] `da_sign_flag` set to 0 indicates a positive sign for the floating-point value. `da_sign_flag` set to 1 indicates a negative sign. The variable `OutSign` is set to equal to `da_sign_flag`. `da_exponent` specifies the exponent of the floating-point value. The value of `da_exponent` should be in the range of 0 to 2^7 - 2 (inclusive). The value 2^7 - 1 is reserved. The decoder should treat the value 2^7 - 1 as indicating an unspecified value. The variable `OutExp` is set to equal to `da_exponent`. `da_mantissa_len_minus1` plus 1 specifies the number of bits in the `da_mantissa` syntax element. The value of `da_mantissa_len_minus1` should be in the range of 0 to 31 (inclusive). The variable `OutManLen` is set to equal to `da_mantissa_len_minus1` + 1. `da_mantissa` specifies the mantissa of the floating-point value. The variable `OutMantissa` is set to equal to `da_mantissa`.

[0089] The Extended DRAP Instruction SEI message is as follows. An example of the Extended DRAP Instruction SEI message syntax is shown below.

[0090]

[0091] An example of the semantics of an Extended DRAP Indicator SEI message is as follows. The picture associated with an Extended DRAP (EDRAP) Indicator SEI message is called an EDRAP picture. The existence of an EDRAP Indicator SEI message indicates that constraints on the picture order and picture references specified in this sub-clause apply. These constraints enable the decoder to correctly decode EDRAP pictures and pictures in the same layer whose decoding and output order both follow the EDRAP picture, without needing to decode any other pictures in the same layer besides the picture referenceablePictures. This includes a list of IRAP or EDRAP pictures within the same CLVS and identified by the edrap_ref_rap_id[i] syntax element in a decoding order.

[0092] All constraints indicated by the presence of the SEI message indicated by EDRAP shall apply, as follows: The EDRAP picture is a trailing picture. The EDRAP picture has a temporal sublayer identifier equal to 0. Except for referenceable Pictures, the EDRAP picture does not include any pictures in the same layer in its active entry in the reference picture list. Except for referenceable Pictures, any picture in the same layer that is decoded and output after the EDRAP picture, in its active entry in the reference picture list, does not include any picture in the same layer that is decoded or output before the EDRAP picture.

[0093] When `edrap_leading_pictures_decodable_flag` equals 1, the following applies: Any picture in the same layer that is decoded after the EDRAP picture should be output after any picture in the same layer that is decoded before the EDRAP picture. Except for `referenceablePictures`, any picture in the same layer that is decoded after the EDRAP picture and output before the EDRAP picture is not included in the active entry of its reference picture list as any picture in the same layer that is decoded before the EDRAP picture. Any picture in the list of `referenceablePictures` is not included in the active entry of its reference picture list as any picture in the same layer that is not a preceding picture in the list of `referenceablePictures`. Therefore, the first picture in `referenceablePictures`, even if it is an EDRAP picture instead of an IRAP picture, does not include any picture from the same layer in the active entry of its reference picture list.

[0094] `edrap_rap_id_minus1` increments by 1 to specify the Random Access Point (RAP) image identifier for the EDRAP image, denoted as `RapPicId`. Each IRAP or EDRAP image is associated with a `RapPicId` value. The `RapPicId` value for an IRAP image is inferred to be equal to 0. Any two EDRAP images associated with the same IRAP image should have different `RapPicId` values. In a bitstream conforming to this disclosure, `edrap_reserved_zero_12bits` should be equal to 0. Other values ​​of `edrap_reserved_zero_12bits` are reserved. The decoder may ignore the value of `edrap_reserved_zero_12bits`. `edrap_num_ref_rap_pics_minus1` increments by 1 to indicate the number of IRAP or EDRAP images within the same CLVS as the EDRAP image and can be included in the active entry of the reference image list for the EDRAP image. `edrap_ref_rap_id[i]` indicates the RapPicId of the i-th RAP image in the active entry of the list of reference images that can be included in the EDRAP image. The i-th RAP image should be an IRAP image associated with the current EDRAP image or an EDRAP image associated with the same IRAP image as the current EDRAP image.

[0095] The following are example technical problems solved by publicly disclosed technical solutions. The example designs for the Annotated Region SEI message, the Depth Representation Information SEI message, and the EDRAP Indication SEI message have at least the following problems. For the Annotated Region SEI message, the range of values ​​for the AR object label index of the ue(v) encoding / decoding syntax element for the i-th Annotated Region object index (ar_object_label_idx[ar_object_idx[i]]) is missing. A practical problem associated with the lack of a specified range of values ​​for the syntax element encoded / decoded for ue(v) is that the designer may be unsure how many bits are available for the corresponding variable during implementation. If the maximum number of bits used in the implementation is insufficient, the decoder may crash when encountering a value greater than the maximum allowed by the number of bits used. For the Depth Representation Information SEI message, the descriptor (e.g., encoding / decoding method) for the i-th Depth Nonlinear Representation Model (depth_nonlinear_representation_model[i]) syntax element is not specified. Without specifying the encoding / decoding method, the decoder may be unable to determine how to parse the syntax element. For the depth representation information (SEI) message, the value ranges for the syntax elements depth representation type (depth_representation_type), disparity reference view identifier (disparity_ref_view_id), depth nonlinear representation number minus one (depth_nonlinear_representation_num_minus1), and depth_nonlinear_representation_model[i] are not specified for ue(v) encoding / decoding. For the EDRAP indication (SEI) message, the semantics of the syntax element edrap_leading_pictures_decodable_flag are missing.

[0096] This document discloses mechanisms for addressing one or more of the problems listed above. For example, this disclosure specifies the range of example values ​​for ar_object_label_idx[ar_object_idx[i]]. Furthermore, this disclosure specifies the range of example descriptors for depth_nonlinear_representation_model[i]. Additionally, this disclosure specifies the range of example values ​​for depth_representation_type, disparity_ref_view_id, depth_nonlinear_representation_num_minus1, and depth_nonlinear_representation_model[i]. Furthermore, this disclosure specifies the example semantics for edrap_leading_pictures_decodable_flag.

[0097] Figure 1 This is a schematic diagram of an example bitstream 100. Bitstream 100 may include compressed video and related syntax. For example, bitstream 100 may be encoded by an encoder, transmitted over one or more networks, and decoded by a decoder for display to a user. For example, bitstream 100 may be defined as a sequence of bits representing a sequence of access units (AUs) that form one or more codec video sequences (CVS). An AU is a collection of one or more pictures associated with a corresponding output time in the video sequence. The bitstream may take the form of a Network Abstraction Layer (NAL) unit stream or a byte stream.

[0098] Bitstream 100 includes one or more Sequence Parameter Sets (SPS) 113, multiple Picture Parameter Sets (PPS) 115, multiple stripes 125, Annotation Region (AR) SEI messages 131, DRI SEI messages 133, and EDRAP Indication SEI messages 135. SPS 113 contains sequence data-related parameters common to all pictures in the encoded / decoded video sequence contained in bitstream 100. Parameters in SPS 113 may include picture size, bit depth, encoding / decoding tool parameters, bit rate limits, etc. It should be noted that although each sequence refers to SPS 113, in some examples, a single SPS 113 may contain data from multiple sequences. PPS 115 contains parameters applied to the entire picture. Therefore, each picture in the video sequence can refer to PPS 115. It should be noted that although each picture refers to PPS 115, in some examples, a single PPS 115 may contain data from multiple pictures. For example, multiple similar pictures may be encoded / decoded based on similar parameters. In this case, a single PPS 115 may contain data from such similar pictures. PPS115 can indicate the encoding / decoding tools, quantization parameters, offsets, etc., that can be used for the stripes in the corresponding image.

[0099] Each slice contains a slice header and image data of a region in the image. The slice header contains parameters specific to each slice. Therefore, each slice in a video sequence may have one slice header. The slice header may contain slice type information, image sequence count (POC), a list of reference images, prediction weights, slice entry points, deblocking parameters, etc. It should be noted that in some examples, bitstream 100 may also include an image header, which is a syntax structure containing parameters applied to all slices in a single image. Therefore, in some contexts, image headers and slice headers can be used interchangeably. For example, certain parameters may be moved between slice headers and image headers, depending on whether these parameters are common to all slices in the image. The image data in slice 125 contains video data encoded according to inter-frame prediction and / or intra-frame prediction, as well as the corresponding transform and quantization residual data. Video data from one or more slices can be encoded and decoded from the image by an encoder and decoded in a decoder to reconstruct the image.

[0100] A stripe 125 can be defined as an integer number of complete slices or an integer number of consecutive complete code-decode tree unit (CTU) rows (e.g., within a slice), where a slice or CTU row is contained within a single NAL unit. Therefore, stripe 125 is also contained within a single NAL unit. Each stripe 125 is further divided into CTUs and / or code-decode tree blocks (CTBs). A CTU is a set of samples of a predefined size that can be segmented using a code-decode tree. A CTB is a subset of CTUs and contains the luma or chroma components of the CTUs. CTUs / CTBs are further divided into code-decode blocks based on the code-decode tree. The code-decode blocks can then be encoded / decoded according to a prediction mechanism.

[0101] Bitstream 100 may include one or more SEI messages. An SEI message is a syntactic structure with defined semantics that conveys information not required by the decoding process in order to determine sample values ​​in the decoded image. Bitstream 100 may include many different SEI messages for different functions. In this example, the bitstream includes AR SEI message 131, DRI SEI message 133, and EDRAP indication SEI message 135.

[0102] AR SEI message 131 is a type of SEI message that carries parameters that identify one or more annotation regions in an image by employing bounding boxes. The bounding boxes represent the size and location of the annotation region and identify one or more objects contained within it. Therefore, AR SEI message 131 contains metadata describing regions in an image. The decoder can use AR SEI message 131 to determine whether and / or how to process these regions during the display process. AR SEI message 131 includes the syntax element ar_object_label_idx[ar_object_idx[i]] 141. ar_object_label_idx[ar_object_idx[i]] 141 indicates the index of the label corresponding to the i-th indexed AR object (the ar_object_idx[i]-th object). For example, AR objects are indexed, and any i-th AR object can be determined by ar_object_idx[i]. Furthermore, AR object labels are indexed, and any i-th AR object label can be determined by ar_object_label_idx[i]. Thus, AR SEI ar_object_label_idx[ar_object_idx[i]]141 obtains the index of the label of the i-th AR object.

[0103] DRI SEI message 133 is an SEI message carrying picture parameters containing depth and / or parallax information for rendering on a three-dimensional (3D) display. Depth is the position of a pixel / sample in 3D space. Parallax is the displacement between the positions of two features (e.g., two pixels) in an image plane. DRI SEI message 133 contains the syntax elements depth_nonlinear_representation_model[i]142, depth_nonlinear_representation_num_minus1143, depth_representation_type144, and disparity_ref_view_id145. depth_nonlinear_representation_model[i]142 specifies each of i piecewise linear segments for mapping the decoded luminance sample values ​​(e.g., depth values) of the auxiliary picture to a scale uniformly quantized according to parallax. depth_nonlinear_representation_num_minus1143+2 specifies the number of piecewise linear segments used to map the depth values ​​to a scale uniformly quantized according to parallax. Therefore, depth_nonlinear_representation_num_minus1 143 plus 2 specifies the number of i segments in depth_nonlinear_representation_model[i]142. depth_representation_type 144 specifies the representation definition of the decoded luminance samples of the auxiliary picture. The allowed values ​​of depth_representation_type 144 and the corresponding explanations for each value are included in Table 1 above and / or Table Y1 below. disparity_ref_view_id 145 specifies the view identifier (ViewId) value from which the disparity value is derived. Therefore, disparity_ref_view_id 145 indicates the ViewId value used as a reference when determining the disparity (e.g., displacement and / or difference between positions) of the samples in the auxiliary picture.

[0104] EDRAP Indication SEI Message 135 indicates the use of EDRAP pictures. An EDRAP picture is a random access picture encoded and decoded using inter-frame prediction based on one or more reference pictures. For example, an EDRAP picture can be encoded and decoded by referencing preceding EDRAP pictures and / or preceding IRAP pictures. An IRAP picture is encoded and decoded using intra-frame prediction and can be decoded without referencing other pictures. An EDRAP scheme can employ an external bitstream containing a set of timing reference pictures for each EDRAP picture. In this way, EDRAP pictures can be selected for random access to the main bitstream, and reference pictures for decoding the EDRAP pictures can be obtained from the external bitstream. EDRAP Indication SEI Message 135 is an SEI message that indicates constraints on picture order and picture references to ensure that the decoder can select any EDRAP picture for random access and successfully decodes the selected EDRAP picture using only the corresponding reference picture (e.g., in the external bitstream). EDRAP Instruction SEI Message 135 includes the syntax element edrap_leading_pictures_decodable_flag146, which contains values ​​indicating whether a set of sorting constraints are applied to the EDRAP picture corresponding to EDRAP Instruction SEI Message 135.

[0105] The presence of EDRAP Indication SEI message 135 imposes certain restrictions on the order of EDRAP images in the bitstream. For example, each EDRAP image is a trailing image. Furthermore, each EDRAP image has a temporal sublayer identifier equal to zero. The temporal sublayer divides the image into a base layer and one or more enhancement layers. Decoders with lower capabilities can decode and display the base layer at a lower frame rate, while decoders with higher capabilities can decode more and more enhancement layers to achieve a higher frame rate. Restricting the temporal sublayer identifier to zero ensures that the EDRAP image is in the base layer and therefore usable by all decoders. Another constraint requires that, in addition to the reference image list, each EDRAP image does not include any images in the same layer from the active entry of the reference image list of the EDRAP image. The reference image list includes IRAP and / or EDRAP images in the decoding order. Therefore, this constraint restricts the EDRAP image to referencing only previous IRAP and EDRAP images. Another constraint requires that, in addition to the list of reference images, any image in the same layer as the EDRAP image and whose decoding order and output order both follow the EDRAP image must not be included in the active entry of that image's list of reference images as any other image in the same layer and whose decoding order or output order precedes the EDRAP image. This constraint prevents images following the EDRAP image from referencing images preceding the EDRAP image. In the case of random access to the EDRAP image, the preceding image will be unavailable; therefore, referencing such an image via a trailing image will result in an error due to the unavailability of the reference image.

[0106] The `edrap_leading_pictures_decodable_flag` syntax element (146) imposes additional constraints on EDRAP pictures. The first of these constraints specifies that any picture in the same layer as the EDRAP picture and decoded after it should be output after any other picture in the same layer as the EDRAP picture and decoded before it. In some cases, the encoding / decoding order and the output order are different. This allows for better compression in some situations, but requires reordering the pictures before display. The picture decoded after the random access point and output before it is called the leading picture. This constraint ensures that the leading picture of an EDRAP picture is not placed before a trailing picture from an EDRAP picture that was output before it.

[0107] The second constraint in this set of additional constraints stipulates that any image on the same layer as the EDRAP image, decoded after the EDRAP image, and output before the EDRAP image, should not include any other image on the same layer as the EDRAP image and decoded before the EDRAP image in the active entry of the image's reference image list, except for the list of referenceable images. This constraint ensures that the leading image only references images after the EDRAP image and IRAP and / or EDRAP images in the list of referenceable images. This ensures that the leading image can be decoded when the corresponding EDRAP image is used for random access.

[0108] As mentioned above, in some example systems, the value ranges, descriptors, and / or semantics of ar_object_label_idx[ar_object_idx[i]]141, depth_nonlinear_representation_model[i]142, depth_nonlinear_representation_num_minus1143, depth_representation_type144, disparity_ref_view_id145, and edrap_leading_pictures_decodable_flag146 are not specified. Therefore, this disclosure includes these value ranges, descriptors, and / or semantics of the aforementioned parameter / syntax elements. This allows the decoder to correctly interpret these values ​​without experiencing undefined behavior such as glitches and / or crashes.

[0109] It should be noted that `ar_object_label_idx[ar_object_idx[i]]` 141, `depth_nonlinear_representation_model[i]` 142, `depth_nonlinear_representation_num_minus1` 143, `depth_representation_type` 144, `disparity_ref_view_id` 145, and `edrap_leading_pictures_decodable_flag` 146 can be associated with descriptors that indicate the encoding / decoding mechanism used to encode and decode the corresponding syntax element. Such descriptors can include `ue(v)`, `u(N)`, `se(v)`, and `u(v)`. `ue(v)` indicates that the syntax element value is encoded and decoded as an unsigned integer exponential Golombian encoding syntax element, where the left bit comes first and the number of bits is variable. The exponential Golombian encoding syntax includes a binary representation of the value as incremented by one, and a leading zero represented as a leading value in a decremented format. `u(N)` indicates that the syntax element value is encoded and decoded as an unsigned integer using N bits. se(v) indicates that the syntax element value is encoded or decoded as a signed integer exponentiation Columbus-coded syntax element, where the left bit comes first and the number of bits is variable. u(v) indicates that the syntax element value is encoded or decoded as an unsigned integer using a variable number of bits.

[0110] To address the aforementioned and other issues, the following summarized methods are disclosed. These items should be considered as examples for explaining general concepts, and not interpreted in a narrow way. Furthermore, these items can be used individually or in combination in any way.

[0111] Example 1

[0112] In one example, to address at least one of the issues listed above, the value of the syntax element `ar_object_label_idx[ar_object_idx[i]]141` is specified to be in the range N to M (inclusive), where N and M are integer values, and N is less than M. In one example, N = 0, M = 255. In the example, the value of `ar_object_label_idx[ar_object_idx[i]]141` is specified in different ranges, such as 0 to 3 (inclusive), 0 to 7 (inclusive), 0 to 15 (inclusive), 0 to 31 (inclusive), 0 to 63 (inclusive), and so on.

[0113] Example 2

[0114] In one example, to address at least one of the issues listed above, the depth_nonlinear_representation_model[i]142 syntax element is specified as ue(v) encoding / decoding.

[0115] Example 3

[0116] In one example, the value of depth_nonlinear_representation_model[i]142 is specified in the range of N to M (inclusive), for example, N = 0 and M = 65535. In another example, the value of depth_nonlinear_representation_model[i]142 is specified in different ranges, for example, 0 to 3 (inclusive), 0 to 7 (inclusive), 0 to 15 (inclusive), 0 to 31 (inclusive), 0 to 63 (inclusive), 0 to 127 (inclusive), 0 to 255 (inclusive), 0 to 511 (inclusive), 0 to 1023 (inclusive), 0 to 2047 (inclusive), 0 to 4095 (inclusive), 0 to 8191 (inclusive), 0 to 16383 (inclusive), and so on.

[0117] Example 4

[0118] In one example, the `depth_nonlinear_representation_model[i]142` syntax element is specified to be encoded using different encoding / decoding methods. In one example, the `depth_nonlinear_representation_model[i]142` syntax element is specified to be encoded using `u(N)`, where N is a positive integer value, such as a value in the range of 2 to 16 (inclusive). In another example, the `depth_nonlinear_representation_model[i]142` syntax element is specified to be encoded using `se(v)`. In yet another example, the `depth_nonlinear_representation_model[i]142` syntax element is specified to be encoded using `u(v)`, with the length in bits specified as, for example, equal to `Log2(MaxNumModes)`, where the variable `MaxNumModes` indicates the maximum number of patterns, and the function `Log2(x)` returns the base-2 logarithm of x.

[0119] Example 5

[0120] In one example, the syntax element depth_nonlinear_representation_num_minus1 143 is specified to be encoded using a different encoding / decoding method than ue(v), such as u(N), u(v), etc.

[0121] Example 6

[0122] In one example, to address at least one of the issues listed above, the value of depth_representation_type144 is specified to be in the range N to M (inclusive), where N and M are integer values, and N is less than M. In one example, N = 0, M = 15. In another example, the value of depth_representation_type144 is specified to be in different ranges, such as 0 to 3 (inclusive), 0 to 7 (inclusive), 0 to 31 (inclusive), 0 to 63 (inclusive), 0 to 127 (inclusive), 0 to 255 (inclusive), etc.

[0123] Example 7

[0124] In one example, to address at least one of the issues listed above, the value of depth_nonlinear_representation_num_minus1 143 is specified to be in the range of 0 to 62 (inclusive). In another example, the value of depth_nonlinear_representation_num_minus1 143 is specified to be in different ranges, such as 0 to 6 (inclusive), 0 to 14 (inclusive), 0 to 30 (inclusive), 0 to 126 (inclusive), 0 to 254 (inclusive), and so on.

[0125] Example 8

[0126] In one example, to address at least one of the issues listed above, the value of disparity_ref_view_id145 is specified in the range of 0 to 1023 (inclusive). In another example, the value of disparity_ref_view_id145 is specified in different ranges, such as 0 to 63 (inclusive), 0 to 127 (inclusive), 0 to 255 (inclusive), 0 to 511 (inclusive), 0 to 2047 (inclusive), 0 to 4095 (inclusive), 0 to 8191 (inclusive), 0 to 16383 (inclusive), 0 to 32767 (inclusive), 0 to 65535 (inclusive), etc.

[0127] Example 9

[0128] In one example, to address at least one of the issues listed above, the semantics of the `edrap_leading_pictures_decodable_flag 146` syntax element are specified as follows: `edrap_leading_pictures_decodable_flag 146` equal to 1 specifies that the following constraints apply: The output order of any picture in the same layer that is decoded after the EDRAP picture should be after any picture in the same layer that is decoded before the EDRAP picture. Except for `referenceablePictures`, any picture in the same layer that is decoded after the EDRAP picture and output before the EDRAP picture should not be included in the active entry of any picture in the same layer that is decoded before the EDRAP picture in its list of reference pictures. `edrap_leading_pictures_decodable_flag 146` equal to 0 does not impose such a constraint.

[0129] An embodiment of the foregoing example will now be described. This embodiment can be applied to VSEI. Most of the relevant parts that have been added or modified relative to the VSEI specification are displayed in bold underlined font, and some deleted parts are displayed in bold italic font. Other editing changes are not highlighted.

[0130] The following is an example of the semantics of the SEI (Separate Interface) message for annotated regions. The SEI message carries parameters that identify the annotated region using a bounding box representing the size and location of the identified object. ...

[0132] `ar_object_label_idx[ar_object_idx[i]]` indicates the index of the label corresponding to the `ar_object_idx[i]`-th object. When `ar_object_label_idx[ar_object_idx[i]]` does not exist, its value is inferred from the SEI message of the annotation region output earlier in the same CVS (if any). ...

[0134] Depth Representation Information SEI Message Syntax

[0135]

[0136]

[0137] An example of the semantics of the Depth Representation Information (SEI) message is as follows. The syntax elements in the Depth Representation Information (DRI) SEI message specify various parameters for an auxiliary picture of type AUX_DEPTH so that the decoded main and auxiliary pictures are processed before rendering on a 3D display, such as in view composition. Specifically, the depth or parallax range of the depth picture is specified. ...

[0139] `depth_representation_type` specifies the representation definition of the decoded luminance samples for the auxiliary image as specified in Table Y1. In Table Y1, parallax specifies the horizontal displacement between two texture views, while the Z value specifies the distance from the camera. The variable maxVal is set to equal to (1 < <BitDepthY)–1。

[0140]

[0141] Definition of table Y1–depth_representation_type

[0142] The disparity_ref_view_id specifies the ViewId value on which the disparity value is derived.

[0143] Disparity_ref_view_id exists only when d_min_flag equals 1 or d_max_flag equals 1, and is useful for depth_representation_type values ​​of 1 and 3. The variables in column x of table Y2 are derived from the corresponding variables in columns s, e, n, and v of table Y2, as follows. If the value of e is in the range of 0 to 127 (excluding the extreme values), then x is set to equal to (-1)s*2e-31*(1+n÷2v). Otherwise (e equals 0), x is set to equal to (-1)s*2-(30+v)*n.

[0144] ZNear ZNearSign ZNearExp ZNearMantissa ZNearManLen ZFar ZFarSign ZFarExp ZFarMantissa ZFarManLen DMax DMaxSign DMaxExp DMaxMantissa DMaxManLen DMin DMinSign DMinExp DMinMantissa DMinManLen

[0145] Table Y2 – Relationship between depth parameter variables and syntax elements

[0146] When present, the dmin and DMax values ​​are specified in units of the luminance sample width of the encoded / decoded image, where ViewId is equal to the ViewId of the auxiliary image. When present, the ZNear and ZFar values ​​have the same units, but are not specified.

[0147] depth_nonlinear_representation_num_minus1 plus 2 specifies the number of piecewise linear segments used to map depth values ​​to a scale uniformly quantized according to disparity. For i, ranging from 0 to depth_nonlinear_representation_num_minus1+2 (inclusive), depth_nonlinear_representation_model[i] specifies the piecewise linear segments used to map the decoded luminance sample values ​​of the auxiliary image to a scale uniformly quantized according to disparity. The values ​​of depth_nonlinear_representation_model[0] and depth_nonlinear_representation_model[depth_nonlinear_representation_num_minus1+2] are both inferred to be equal to 0. ...

[0149] An example of the semantics of an Extended DRAP Indicator SEI message is as follows. The picture associated with an Extended DRAP (EDRAP) Indicator SEI message is called an EDRAP picture. The existence of the EDRAP Indicator SEI message indicates that the constraints on picture order and picture references specified in this sub-clause apply. These constraints enable the decoder to correctly decode EDRAP pictures and pictures in the same layer whose decoding and output order both follow the EDRAP picture, without needing to decode any other pictures in the same layer besides the picture referenceablePictures. This includes a list of IRAP or EDRAP pictures within the same CLVS and identified by the edrap_ref_rap_id[i] syntax element with a decoding order.

[0150] All constraints indicated by the presence of the SEI message indicated by EDRAP shall apply, as follows: The EDRAP picture is a trailing picture. The EDRAP picture has a temporal sublayer identifier equal to 0. Except for referenceable Pictures, the EDRAP picture does not include any pictures in the same layer in its active entry in the reference picture list. Except for referenceable Pictures, any picture in the same layer that is decoded and output after the EDRAP picture, in its active entry in the reference picture list, does not include any picture in the same layer that is decoded or output before the EDRAP picture.

[0151]

[0152] Any image in the list of referenceablePictures is not included in the active entry of its reference image list if it is not in the same layer as any image in the list of referenceablePictures. Therefore, the first image in referenceablePictures, even if it is an EDRAP image instead of an IRAP image, does not include any images from the same layer in the active entry of its reference image list.

[0153] The increment of 1 in `edrap_rap_id_minus1` specifies the RAP image identifier for the EDRAP image, represented as `RapPicId`. Each IRAP or EDRAP image is associated with a `RapPicId` value. The `RapPicId` value for an IRAP image is inferred to be equal to 0. Any two EDRAP images associated with the same IRAP image should have different `RapPicId` values.

[0154]

[0155] In this version of the bitstream of this specification, `edrap_reserved_zero_12bits` should be equal to 0. Other values ​​of `edrap_reserved_zero_12bits` are reserved. The decoder should ignore the value of `edrap_reserved_zero_12bits`. `edrap_num_ref_rap_pics_minus1` plus 1 indicates the number of IRAP or EDRAP pictures within the same CLVS as the EDRAP picture, and the number of active entries that can be included in the reference picture list of the EDRAP picture. `edrap_ref_rap_id[i]` indicates the RapPicId of the i-th RAP picture that can be included in the active entry of the reference picture list of the EDRAP picture. The i-th RAP picture should be either an IRAP picture associated with the current EDRAP picture or an EDRAP picture associated with the same IRAP picture as the current EDRAP picture. ...

[0157] Figure 2This is a block diagram of an example video processing system 4000 that can implement various technologies disclosed herein. Various implementations may include some or all of the components in system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8 or 10-bit multi-component pixel values), or in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces (such as Ethernet, Passive Optical Networking (PON), etc.) and wireless interfaces (such as Wi-Fi or cellular interfaces).

[0158] System 4000 may include an encoding / decoding component 4004 capable of implementing the various encoding / decoding or coding methods described in this document. Encoding / decoding component 4004 can reduce the average bit rate of the video from input 4002 to the output of encoding / decoding component 4004 to produce an encoded / decoded representation of the video. Therefore, encoding / decoding techniques are sometimes referred to as video compression or video transcoding techniques. The output of encoding / decoding component 4004 can be stored or transmitted via connected communication, as represented by component 4006. The stored or communicated bitstream (or encoded / decoded) representation of the video received at input 4002 can be used by component 4008 to generate pixel values ​​or displayable video that is sent to display interface 4010. The process of generating user-visible video from the bitstream representation is sometimes referred to as video decompression. Furthermore, although some video processing operations are referred to as “encoding / decoding” operations or tools, it should be understood that encoding / decoding tools or operations are used at the encoder, and the corresponding decoding tools or operations for retrieving the results of encoding / decoding are used at the decoder.

[0159] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), or DisplayPort. Examples of storage interfaces include SATA (Serial Advanced Technology Accessory), PCI, IDE, etc. The technologies described in this document can be implemented in a variety of electronic devices, such as mobile phones, laptops, smartphones, or other devices capable of digital data processing and / or video display.

[0160] Figure 3This is a block diagram of an example video processing apparatus 4100. Apparatus 4100 can be used to implement one or more methods described herein. Apparatus 4100 can be implemented in smartphones, tablets, computers, Internet of Things (IoT) receivers, etc. Apparatus 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. The processors(multiple) 4102 may be configured to implement one or more methods described herein. The memories(multiple) 4104 may be used to store data and code for implementing the methods and techniques described herein. The video processing circuitry 4106 may be used to implement some of the techniques described herein in hardware circuitry. In some embodiments, the video processing circuitry 4106 may be at least partially included within the processor 4102, such as a graphics coprocessor.

[0161] Figure 4 This is a flowchart of an example method 4200 for video processing. Method 4200 includes: in step 4202, determining that the value of the ar_object_label_idx[ar_object_idx[i]] syntax element is specified within the range N to M. N and M are integers, and N is less than M. For example, N can be 0, M can be 3, 7, 15, 31, 63, or 255, and the range N to M can include end values. In the example, the ar_object_label_idx[ar_object_idx[i]] syntax element is included in the AR SEI message.

[0162] In step 4204, the value of the `depth_representation_type` syntax element is determined. The `depth_representation_type` syntax element can be specified within the range A to B. A and B can be integers, where A is less than B. For example, A can be 0, and B can be 3, 7, 15, 31, 63, 127, or 255, and the range A to B can include end values. In the example, the `depth_representation_type` syntax element is included in the DRI SEI message.

[0163] In step 4206, the value of the `depth_nonlinear_representation_model[i]` syntax element is determined. The `depth_nonlinear_representation_model[i]` syntax element is specified within the range C to D. C and D can be integers, and C is less than D. For example, C can be 0, D can be 6, 14, 30, 62, 126, or 254, and the range C to D can include end values. In the example, the `depth_nonlinear_representation_model[i]` syntax element is included in the DRISEI message.

[0164] In step 4208, the value of the disparity_ref_view_id syntax element is determined. The disparity_ref_view_id syntax element is specified within the range E to F. E and F can be integers, where E is less than F. For example, E can be 0, and F can be 63, 127, 255, 511, 1023, 2047, 4095, 8191, 16383, 32767, or 65535, and the range E to F can include end values. In the example, the disparity_ref_view_id syntax element is included in the DRI SEI message.

[0165] In step 4210, a conversion between visual media data and a bitstream is performed based on ar_object_label_idx[ar_object_idx[i]], depth_nonlinear_representation_model[i], disparity_ref_view_id, and / or disparity_ref_view_id. When method 4200 is performed on the encoder, the conversion includes encoding the visual media data into a bitstream. When method 4200 is performed on the decoder, the conversion includes decoding the bitstream to obtain the visual media data.

[0166] It should be noted that method 4200 can be implemented in an apparatus for processing video data, the apparatus including a processor and a non-transitory memory thereon with instructions, such as a video encoder 4400, a video decoder 4500, and / or an encoder 4600. In this case, the instructions executed by the processor cause the processor to perform method 4200. Alternatively, method 4200 can be executed by a non-transitory computer-readable medium comprising a computer program product for use by a video codec device. This computer program product includes computer-executable instructions stored on the non-transitory computer-readable medium, such that the computer-executable instructions, when executed by a processor, cause the video codec device to perform method 4200.

[0167] Figure 5 This is a block diagram illustrating an example video encoding / decoding system 4300 from which the technology of this disclosure may be utilized. The video encoding / decoding system 4300 may include a source device 4310 and a target device 4320. The source device 4310 generates encoded video data, which may be referred to as a video encoding device. The target device 4320 can decode the encoded video data generated by the source device 4310, and this target device 4320 may be referred to as a video decoding device.

[0168] Source device 4310 may include video source 4312, video encoder 4314, and input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system that generates video data, or a combination of these sources. Video data may include one or more pictures. Video encoder 4314 encodes the video data from video source 4312 to generate a bitstream. The bitstream may include a sequence of bits forming a codec representation of the video data. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax elements. I / O interface 4316 includes a modulator / demodulator (modem) and / or a transmitter. Encoded video data may be transmitted directly to target device 4320 via network 4330 through I / O interface 4316. Encoded video data may also be stored on storage medium / server 4340 for access by target device 4320.

[0169] Target device 4320 may include I / O interface 4326, video decoder 4324, and display device 4322. I / O interface 4326 may include a receiver and / or a modem. I / O interface 4326 may acquire encoded video data from source device 4310 or storage medium / server 4340. Video decoder 4324 may decode the encoded video data. Display device 4322 may display the decoded video data to a user. Display device 4322 may be integrated with target device 4320 or may be external to target device 4320 configured to connect to an external display device.

[0170] The video encoder 4314 and the video decoder 4324 can operate according to video compression standards such as High Efficiency Video Codec (HEVC), Multi-Functional Video Codec (VVC), and other current and / or other standards.

[0171] Figure 6 This is a block diagram illustrating an example of a video encoder 4400, which can be... Figure 5The system 4300 shown includes a video encoder 4314. The video encoder 4400 can be configured to perform any or all of the techniques disclosed herein. The video encoder 4400 includes multiple functional components. The techniques described herein can be shared among the various components of the video encoder 4400. In some examples, a processor can be configured to perform any or all of the techniques described herein.

[0172] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405, and an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a buffer 4413, and an entropy coding unit 4414.

[0173] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In one example, the prediction unit 4402 may include an intra-block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is the picture in which the current video block is located.

[0174] Furthermore, some components such as the motion estimation unit 4404 and the motion compensation unit 4405 can be highly integrated, but for illustrative purposes they are shown separately in the example of the video encoder 4400.

[0175] The segmentation unit 4401 can segment an image into one or more video blocks. The video encoder 4400 and the video decoder 4500 can support various video block sizes.

[0176] The mode selection unit 4403 can, for example, select one of the intra-frame or inter-frame encoding / decoding modes based on the error result, and provide the obtained intra-frame or inter-frame encoded / decoded blocks to the residual generation unit 4407 to generate residual block data and to the reconstruction unit 4412 to reconstruct the encoded / decoded blocks for use as reference images. In some examples, the mode selection unit 4403 can select a combined intra-frame and inter-frame prediction (CIIP) mode, where prediction is based on inter-frame prediction signals and intra-frame prediction signals. The mode selection unit 4403 can also select the resolution of the motion vector (e.g., sub-pixel or integer pixel precision) for blocks in the inter-frame prediction case.

[0177] To perform inter-frame prediction for the current video block, motion estimation unit 4404 can generate motion information for the current video block by comparing one or more reference frames from buffer 4413 with the current video block. Motion compensation unit 4405 can determine the predicted video block for the current video block based on the motion information of the image from buffer 4413 (rather than the image associated with the current video block) and decoded samples.

[0178] The motion estimation unit 4404 and the motion compensation unit 4405 can perform different operations on the current video block. For example, the different operations performed depend on whether the current video block is in an I-strip, a P-strip, or a B-strip.

[0179] In some examples, motion estimation unit 4404 can perform unidirectional prediction of the current video block, and can search for a reference video block for the current video block in the reference images of list 0 or list 1. Motion estimation unit 4404 can then generate a reference index indicating that the reference video block is present in the reference images of list 0 or list 1, and a motion vector indicating the spatial displacement between the current video block and the reference video block. Motion estimation unit 4404 can output the reference index, prediction direction indicator, and motion vector as motion information for the current video block. Motion compensation unit 4405 can generate a predicted video block for the current block based on the reference video block indicated by the motion information of the current video block.

[0180] In other examples, motion estimation unit 4404 can perform bidirectional prediction of the current video block. Motion estimation unit 4404 can search for a reference video block for the current video block in the reference images of list 0 and can also search for another reference video block for the current video block in the reference images of list 1. Motion estimation unit 4404 can then generate a reference index indicating that the reference images in list 0 or list 1 contain the reference video block, and a motion vector indicating the spatial displacement between the reference video block and the current video block. Motion estimation unit 4404 can output the reference index and the motion vector of the current video block as the motion information of the current video block. Motion compensation unit 4405 can generate a predicted video block for the current video block based on the reference video block indicated by the motion information of the current video block.

[0181] In some examples, the motion estimation unit 4404 may output the complete set of motion information for decoding processing by the decoder. In some examples, the motion estimation unit 4404 may not output the complete set of motion information for the current video. Instead, the motion estimation unit 4404 may refer to the motion information of another video block to signal the motion information of the current video block. For example, the motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of adjacent video blocks.

[0182] In one example, the motion estimation unit 4404 may indicate in the syntax structure associated with the current video block that the video decoder 4500 indicates that the current video block has the same motion information value as another video block.

[0183] In another example, motion estimation unit 4404 can identify another video block and motion vector difference (MVD) in the syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicating video block. Video decoder 4500 can use the motion vector of the indicating video block and the motion vector difference to determine the motion vector of the current video block.

[0184] As discussed above, the video encoder 4400 can predictively signal motion vectors. Two examples of predictive signaling notification techniques that can be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and merge pattern signaling notification.

[0185] The intra-prediction unit 4406 can perform intra-prediction on the current video block. When performing intra-prediction on the current video block, the intra-prediction unit 4406 can generate prediction data for the current video block based on the decoded samples of other video blocks in the same frame. The prediction data for the current video block may include the predicted video block and various syntax elements.

[0186] The residual generation unit 4407 can generate residual data for the current video block by subtracting (or more) predicted video blocks from the current video block. The residual data for the current video block can include residual video blocks corresponding to different sample components in the current video block.

[0187] In other examples, such as in skip mode, residual data for the current video block may not exist, and the residual generation unit 4407 may not perform a subtraction operation.

[0188] The transform processing unit 4408 can generate one or more transform coefficient video blocks of the current video block by applying one or more transforms to the residual video block associated with the current video block.

[0189] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 can quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0190] The inverse quantization unit 4410 and the inverse transform unit 4411 can apply inverse quantization and inverse transform to the transform coefficient video block respectively to reconstruct the residual video block from the transform coefficient video block. The reconstruction unit 4412 can add the reconstructed residual video block to the corresponding sample points of one or more predicted video blocks generated by the prediction unit 4402 to produce a reconstructed video block associated with the current block for storage in the buffer 4413.

[0191] After the video block is reconstructed by reconstruction unit 4412, a loop filtering operation can be performed to reduce video block artifacts in the video block.

[0192] The entropy encoding unit 4414 can receive data from other functional components of the video encoder 4400. When the entropy encoding unit 4414 receives data, it can perform one or more entropy encoding operations to generate entropy-encoded data and output a bitstream including the entropy-encoded data.

[0193] Figure 7 This is a block diagram illustrating an example of a video decoder 4500, which can be... Figure 5 The system 4300 shown includes a video decoder 4324. The video decoder 4500 can be configured to perform any or all of the techniques disclosed herein. In the example shown, the video decoder 4500 includes multiple functional components. The techniques described herein can be shared among the various components of the video decoder 4500. In some examples, the processor can be configured to perform any or all of the techniques described herein.

[0194] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra-frame prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 can perform a decoding process that is the overall inversion of the encoding process described with respect to the video encoder 4400.

[0195] The entropy decoding unit 4501 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded blocks of video data). The entropy decoding unit 4501 can decode the entropy-encoded video, and based on the entropy-encoded video data, the motion compensation unit 4502 can determine motion information including motion vectors, motion vector precision, reference image list index, and other motion information. The motion compensation unit 4502 can determine this information, for example, by performing AMVP and merge modes.

[0196] The motion compensation unit 4502 can generate motion compensation blocks, possibly based on interpolation filters. The identifier of the interpolation filter to be used at sub-pixel precision can be included in the syntax element.

[0197] The motion compensation unit 4502 can use the interpolation filter used by the video encoder 4400 during the encoding of the video block to calculate the interpolation values ​​of a sub-integer number of pixels of the reference block. The motion compensation unit 4502 can determine the interpolation filter used by the video encoder 4400 based on the received syntax information and use the interpolation filter to generate the prediction block.

[0198] The motion compensation unit 4502 can use some syntactic information to determine: the size of the blocks used to encode (multiple) frames and / or (multiple) stripes of the encoded video sequence, segmentation information describing how each macroblock of the image of the encoded video sequence is segmented, a mode indicating how each segment is encoded, one or more reference frames (and a list of reference frames) for each inter-frame coded block, and other information for decoding the encoded video sequence.

[0199] Intra-prediction unit 4503 can use, for example, an intra-prediction mode received in the bitstream to form prediction blocks from spatially adjacent blocks. Inverse quantization unit 4504 inverse-quantizes (i.e., dequantizes) the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 4501. Inverse transform unit 4505 applies the inverse transform.

[0200] The reconstruction unit 4506 can sum the residual blocks using the corresponding prediction blocks generated by the motion compensation unit 4502 or the intra-frame prediction unit 4503 to form a decoded block. As desired, a deblocking filter can also be applied to filter the decoded block to remove block artifacts. The decoded video block is then stored in a buffer 4507, which provides a reference block for subsequent motion compensation / intra-frame prediction and also generates the decoded video for presentation on a display device.

[0201] Figure 8This is a schematic diagram of an example encoder 4600. Encoder 4600 is suitable for implementing VVC technology. Encoder 4600 includes three loop filters: a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike DF 4602, which uses predefined filters, SAO 4604 and ALF 4606 utilize the original samples of the current image and reduce the mean square error between the original and reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, using auxiliary information signaling from the encoder and decoder to inform the offset and filter coefficients. ALF 4606 is located in the final processing stage of each image and can be considered as a tool to attempt to capture and repair artifacts generated by previous stages.

[0202] The encoder 4600 also includes an intra-frame prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610, configured to receive input video. The intra-frame prediction component 4608 is configured to perform intra-frame prediction, while the ME / MC component 4610 is configured to perform inter-frame prediction using a reference picture obtained from a reference picture buffer 4612. Residual blocks from inter-frame or intra-frame prediction are fed into a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are then fed into an entropy codec component 4618. The entropy codec component 4618 entropy codes the prediction results and the quantized transform coefficients and sends them to a video decoder (not shown). The quantization component output from the quantization component 4616 can be fed into an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. REC component 4624 can output images to DF 4602, SAO 4604 and ALF 4606 for filtering before these images are stored in reference image buffer 4612.

[0203] Below is a list of some example preferred solutions.

[0204] The following solutions illustrate examples of the techniques discussed in this article.

[0205] 1. A media data processing method (e.g., Figure 4 The method 4200 described in the document includes: performing a conversion between a video and a bitstream of the video according to a rule, wherein the rule specifies a range of values ​​for one or more syntax fields that indicate the index of a tag of a corresponding object in an annotation region of the video, wherein the range is between N and M, where N and M are integers.

[0206] 2. The method described in Solution 1, wherein N = 0 and M = 255.

[0207] 3. The method described in Solution 1, wherein N = 0 and M = 3.

[0208] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 2).

[0209] 4. A method for processing video data, comprising: performing a conversion between a video and a bitstream of the video according to rules, wherein the rules specify the type of encoding / decoding for encoding / decoding syntax elements, the syntax elements indicating the number of nonlinear segments used in a segmented nonlinear mapping of depth information of objects in the video being encoded / decoded in the bitstream.

[0210] 5. The method according to Solution 1, wherein the rule specifies that the syntax element is encoded as a syntax element of an unsigned integer zero-order exponent Columbus-encoded left-bit-first encoded unsigned integer.

[0211] 6. The method according to Solution 1, wherein the rule specifies that the syntax element is u(N') encoded / decoded, where N' is a positive integer.

[0212] 7. The method according to Solution 1, wherein the rule specifies that the syntax element is encoded as a left-first signed integer zero-order exponent Columbus-coded syntax element.

[0213] 8. The method according to solutions 1-4, wherein the rule specifies that the value of the syntax element is constrained to be within N to M, where N and M are integers.

[0214] 9. The method described in Solution 5, wherein N = 0 and M = 65535.

[0215] 10. The method described in Solution 2, wherein N = 0 and M = 3.

[0216] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 3).

[0217] 11. A method for processing video data, comprising: performing a conversion between a video and a bitstream of the video according to rules, wherein the rules specify constraints on syntactic elements in a supplemental enhancement information syntactic structure representing depth information of one or more objects in the video.

[0218] 12. The method according to solution 11, wherein the syntax element includes a depth representation type, and wherein the rule specifies that the value of the syntax element is constrained to a range of N to M, where N and M are integers.

[0219] 13. The method according to solution 11, wherein the syntax element indicates the number of nonlinear mapping models of depth information, and wherein the rule specifies that the syntax element is in the range of 0 to M, where M is an integer.

[0220] 14. The method according to solution 11, wherein the syntax element indicates an identifier of a parallax reference view, and wherein the rule specifies that the value of the syntax element is between 0 and M, where M is an integer.

[0221] The following solutions show example implementations of the techniques discussed in the previous chapter (e.g., Project 4).

[0222] 15. A method for processing video data, comprising: performing a conversion between a video and a bitstream of the video according to rules, wherein the rules specify: values ​​of flags indicating images as extended dependent random access points, controlling (1) first-order constraints on images in the same layer as the image and whose decoding and output orders follow the image, and (2) second-order constraints on images in the same layer as the image and whose decoding and output orders follow the image.

[0223] 16. The method according to solution 15, wherein the value is equal to 1.

[0224] 17. The method according to any one of solutions 1-16, wherein the conversion includes generating the bitstream from the video.

[0225] 18. The method according to any one of solutions 1-16, wherein the conversion includes generating the video from the bitstream.

[0226] 19. A video decoding apparatus, including a processor, configured to implement the method in one or more of solutions 1 to 18.

[0227] 20. A video encoding apparatus, including a processor, configured to implement the method in one or more of solutions 1 to 18.

[0228] 21. A computer program product having computer code stored thereon, said computer code, when executed by a processor, causing the processor to implement the method of any one of solutions 1 to 18.

[0229] 22. A video processing method comprising generating a bitstream according to any one or more of the methods described in solutions 1-8, and storing the bitstream on a computer-readable medium.

[0230] 23. The methods, apparatus or systems described in this document.

[0231] In this document, the term "video processing" can refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm can be applied during the conversion from a pixel representation of a video to a corresponding bitstream representation, and vice versa. For example, as defined by the syntax, the bitstream representation of the current video block can correspond to bits that are co-located or distributed at different positions within the bitstream. For example, macroblocks can be encoded based on the error residual values ​​of the transform and encoding / decoding, and bits can also be used in the header and other fields of the bitstream. Furthermore, during the conversion, the decoder can, based on this determination, parse the bitstream knowing that some fields may or may not be present, as described in the solutions above. Similarly, the encoder can determine whether to include or exclude certain syntax fields and generate the codec representation accordingly by including or excluding syntax fields from the codec representation.

[0232] The disclosures and other solutions, examples, embodiments, modules, and functional operations described in this document can be implemented in digital electronic circuits, or computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or combinations thereof. The disclosures and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-volatile computer-readable medium for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, a material composition that influences machine-readable propagated signals, or one or more of these. The terms "data processing unit" or "data processing apparatus" include all means, devices, and machines for processing data, including, for example, programmable processors, computers, or multiprocessors or computer groups. In addition to hardware, the apparatus may also include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or combinations thereof. The propagated signals are artificially generated signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device.

[0233] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to that program, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed and executed on one or more computers located at a single site or distributed across multiple sites interconnected by a communication network.

[0234] The processing and logic flows described in this document can be executed by one or more programmable processors that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flows can also be executed by special-purpose logic circuitry, and the devices can be implemented as special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits).

[0235] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor that executes instructions and one or more storage devices that store the instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to mass storage devices, or both. However, a computer does not necessarily have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and optical disc read-only memory (CD ROM) and digital versatile optical disc read-only memory (DVD-ROM). The processor and memory may be supplemented by dedicated logic circuitry or incorporated into dedicated logic circuitry.

[0236] While this patent document contains numerous details, it should not be construed as limiting the scope of any invention or claim, but rather as a description of features of specific embodiments of a particular invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various functions described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. Furthermore, although the foregoing features may be described as functioning in certain combinations, or even initially claimed to be so, in certain circumstances, one or more features from a combination of claims may be removed from the combination, and a combination of claims may refer to a sub-combination or a variation of a sub-combination.

[0237] Similarly, although the operations are described in a specific order in the accompanying drawings, this should not be construed as requiring the specific order or sequence shown to perform such operations, or all the described operations, in order to obtain the desired result. Furthermore, the separation of various system components in the embodiments of this patent document should not be construed as requiring such separation in all embodiments.

[0238] Only some implementations and examples are described; other implementations, enhancements, and variations can be made based on the content described and illustrated in this patent document.

[0239] When no intermediate component exists other than a line, trace, or other medium between the first and second components, the first component is directly coupled to the second component. When an intermediate component other than a line, trace, or other medium exists between the first and second components, the first component is indirectly coupled to the second component. The term "coupling" and its variations include direct coupling and indirect coupling. Unless otherwise stated, the term "about" is used to mean a range including upper and lower 10% of the value.

[0240] While several embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. The present examples are intended to be illustrative rather than restrictive and are not limited to the details given. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented.

[0241] Furthermore, without departing from the scope of this disclosure, the discrete or individual technologies, systems, subsystems, and methods described and illustrated in the various embodiments may be combined or integrated with other systems, modules, technologies, or methods. Other items shown or discussed as coupled may be directly connected or indirectly coupled or communicated via some interface, device, or intermediate component in an electrical, mechanical, or other manner. Those skilled in the art can identify other examples of changes, substitutions, and modifications without departing from the spirit and scope of this disclosure.

Claims

1. A method for processing video data, comprising: Perform the conversion between the video and the video bitstream according to the rules. The rule specifies that the value of the first syntax element in the Deep Representation Information Supplemental Enhancement Information (DRI SEI) message is in the range of 0 to 15, including end values; and The first syntax element in the DRI SEI message specifies the representation definition of the decoded luminance samples of the auxiliary image; The rule states that when the value of the fourth syntax element indicates that a syntax element specifying the minimum disparity value exists in the DRI syntax structure, or when the value of the fifth syntax element indicates that a syntax element specifying the maximum disparity value exists in the DRI syntax structure, the third syntax element indicating the view identifier for which the disparity value is derived is included in the DRI SEI message.

2. The method according to claim 1, wherein, When the value of the first syntax element is equal to 3, the second syntax element is included in the DRI SEI message, and The value of the second syntax element plus 2 specifies the number of piecewise linear segments used to map the depth value to a scale uniformly quantized according to disparity.

3. The method according to claim 2, wherein, The rule specifies that the value of the second syntax element in the DRI SEI message is in the range of 0 to 62, including end values.

4. The method according to claim 1, wherein, The value of the third syntax element in the DRI SEI message is in the range of 0 to 1023, including end values.

5. The method according to claim 1, wherein, The third syntax element is useful for the values ​​of the first syntax element that are equal to 1 and 3.

6. The method according to claim 1, wherein, The rule stipulates that the value of the sixth syntax element in the Augmentation and Enhancement Information (AR SEI) message in the annotation region is in the range of N to M, inclusive, where N and M are integers, and N is less than M. In the AR SEI message, the sixth syntax element indicates the index of the label corresponding to the index of the i-th AR object.

7. The method according to claim 6, wherein, N=0, M=255.

8. The method according to claim 2, wherein, The first syntax element and the second syntax element are exponential Columbus encoding and decoding using unsigned integers.

9. The method according to claim 1, wherein, The third syntax element is an exponential Columbus codec using unsigned integers.

10. The method according to claim 6, wherein, The sixth syntax element is an exponential Columbus codec using unsigned integers.

11. The method according to claim 1, wherein, The conversion includes encoding the video into the bitstream.

12. The method according to claim 1, wherein, The conversion includes decoding the video from the bitstream.

13. An apparatus for processing video data, comprising a processor and a non-transitory memory having instructions thereon, wherein, When the instruction is executed by the processor, the processor: Perform the conversion between the video and the video bitstream according to the rules. The rule specifies that the value of the first syntax element in the Deep Representation Information Supplemental Enhancement Information (DRI SEI) message is in the range of 0 to 15, including end values; The first syntax element in the DRI SEI message specifies the representation definition of the decoded luminance samples of the auxiliary image; The rule states that when the value of the fourth syntax element indicates that a syntax element specifying the minimum disparity value exists in the DRI syntax structure, or when the value of the fifth syntax element indicates that a syntax element specifying the maximum disparity value exists in the DRI syntax structure, the third syntax element indicating the view identifier for which the disparity value is derived is included in the DRI SEI message.

14. The apparatus according to claim 13, wherein, When the value of the first syntax element is equal to 3, the second syntax element is included in the DRI SEI message. Wherein, the value of the second syntax element plus 2 specifies the number of piecewise linear segments used to map the depth value to a scale uniformly quantized according to disparity, and The rule specifies that the value of the second syntax element in the DRI SEI message is in the range of 0 to 62, including end values.

15. The apparatus according to claim 14, in, The value of the third syntax element in the DRI SEI message is in the range of 0 to 1023, including end values; The third syntax element is useful for the values ​​of the first syntax element that are equal to 1 and 3; The rule specifies that the value of the sixth syntax element in the Augmentation and Enhancement Information (AR SEI) message in the annotation region is in the range of N to M, including end values, where N and M are integers, and N is less than M. Wherein, the sixth syntax element in the AR SEI message indicates the index of the label corresponding to the i-th AR object index, and Where N=0, M=255.

16. A non-transitory computer-readable storage medium for storing instructions, said instructions causing a processor to: Perform the conversion between the video and the video bitstream according to the rules. in, The rule specifies that the value of the first syntax element in a Deep Representation Information Supplemental Enhancement Information (DRI SEI) message is in the range of 0 to 15, including end values; The first syntax element in the DRI SEI message specifies the representation definition of the decoded luminance samples of the auxiliary image; The rule states that when the value of the fourth syntax element indicates that a syntax element specifying the minimum disparity value exists in the DRI syntax structure, or when the value of the fifth syntax element indicates that a syntax element specifying the maximum disparity value exists in the DRI syntax structure, the third syntax element indicating the view identifier for which the disparity value is derived is included in the DRI SEI message.

17. The non-transitory computer-readable storage medium according to claim 16, wherein, When the value of the first syntax element is equal to 3, the second syntax element is included in the DRI SEI message. The value of the second syntax element plus 2 specifies the number of piecewise linear segments used to map the depth value to a scale uniformly quantized according to disparity. The rule specifies that the value of the second syntax element in the DRI SEI message is in the range of 0 to 62, including end values; The value of the third syntax element in the DRI SEI message is in the range of 0 to 1023, including end values; The third syntax element is useful for the values ​​of the first syntax element that are equal to 1 and 3; The rule specifies that the value of the sixth syntax element in the Augmentation and Enhancement Information (AR SEI) message in the annotation region is in the range of N to M, including end values, where N and M are integers, and N is less than M. Wherein, the sixth syntax element in the AR SEI message indicates the index of the label corresponding to the i-th AR object index, and Where N=0, M=255.

18. A non-transitory computer-readable recording medium storing a bitstream of video, and further storing a computer program thereon, wherein, When the computer program is executed by a processor, it implements the method of claim 1 to generate the bit stream.

19. The non-transitory computer-readable recording medium according to claim 18, wherein, When the value of the first syntax element is equal to 3, the second syntax element is included in the DRI SEI message. The value of the second syntax element plus 2 specifies the number of piecewise linear segments used to map the depth value to a scale uniformly quantized according to disparity. The rule specifies that the value of the second syntax element in the DRI SEI message is in the range of 0 to 62, including end values; The value of the third syntax element in the DRI SEI message is in the range of 0 to 1023, including end values; The third syntax element is useful for the values ​​of the first syntax element that are equal to 1 and 3; The rule specifies that the value of the sixth syntax element in the Augmentation and Enhancement Information (AR SEI) message in the annotation region is in the range of N to M, including end values, where N and M are integers, and N is less than M. Wherein, the sixth syntax element in the AR SEI message indicates the index of the label corresponding to the i-th AR object index, and Where N=0, M=255.

20. A method for storing a bitstream of video, comprising: The bit stream is generated by performing the method according to any one of claims 1-11; as well as The bit stream is stored in a non-transitory computer-readable recording medium.

Images