Methods, systems, and computer programs for supporting mixed NAL unit types in coded pictures.

Abstract

To provide methods and systems for decoding video streams.SOLUTION: A method includes receiving a first network abstraction layer (NAL) unit of a first slice of a coded picture and a second VCL NAL unit of a second slice of the coded picture, the first VCL NAL unit having a first VCL NAL unit type and the second VCL NAL unit having a second VCL NAL unit type that is different from the first VCL NAL unit type. The method also includes decoding the coded picture, the decoding including determining a picture type of the coded picture based on the first VCL NAL unit type of the first VCL NAL unit and the second VCL NAL unit type of the second VCL NAL unit, or based on an indicator, received by the at least one processor, indicating that the coded picture includes mixed VCL NAL unit types.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 62 / 956,254, filed on January 1, 2020, and U.S. Patent Application No. 17 / 077,035, filed on October 22, 2020, the entire contents of which are incorporated herein by reference.

[0002] Embodiments of the present disclosure relate to video encoding and decoding, and more specifically, to the support for the mixed network abstraction layer (NAL) unit type for coded pictures (coded pictures).

Background Art

[0003] In the draft of the Versatile Video Coding (VVC) specification JVET - P2001 (incorporated herein by reference in its entirety, edited and updated by JVET - Q0041), the function of the mixed network abstraction layer (NAL) unit type is supported, which enables a NAL unit to have one or more slice NAL units with a NAL unit type equal to an Intra Random Access Point (IRAP) or a Clean Random Access (CRA), and one or more slice NAL units with a NAL unit type equal to a non - IRAP. This function can be used to merge two different bitstreams into one or support different random access periods for each local area (sub - picture). Currently, the following syntax and semantics are defined to support the function.

[0004] In Table 1 below, an exemplary picture parameter set unprocessed byte sequence payload (RBSP) syntax is provided. Table 1

Table 1

[0005] A syntax element mixed_nalu_types_in_pic_flag equal to 1 specifies that each picture referencing a Picture Parameter Set (PPS) has more than one Video Coding Layer (VCL) NAL unit, that VCL NAL units do not have the same nal_unit_type, and that the picture is not an IRAP picture. A syntax element mixed_nalu_types_in_pic_flag equal to 0 specifies that each picture referencing a PPS has one or more VCL NAL units, and that the VCL NAL units of each picture referencing a PPS have the same nal_unit_type.

[0006] If the syntax element no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, then the value of the syntax element mixed_nalu_types_in_pic_flag is equal to 0.

[0007] According to the current VVC specification, NAL unit type codes and NAL unit type classes are defined as shown in Table 2 below. Table 2 [Table 2] JPEG0007880407000003.jpg110161

[0008] For each slice in picture picA that has a nal_unit_type value nalUnitTypeA within the range of IDR_W_RADL or CRA_NUT including both ends (picA also includes one or more slices with different nal_unit_type values), the following applies:

[0009] (A) The slice belongs to the subpicture subpicA, and the value of the corresponding syntax element subpic_treated_as_pic_flag[i] is equal to 1.

[0010] (B) A slice shall not belong to a subpicture of picA that contains a VCL NAL unit having a syntax element nal_unit_type that is not equal to nalUnitTypeA.

[0011] (C) In an encoded layer video sequence (CLVS), for all subsequent PUs in decoding order, neither RefPicList[0] nor RefPicList[1] of the slice of subpicA shall contain any picture in the preceding picA in decoding order as an active entry.

[0012] For any specific picture's VCL NAL unit, apply the following:

[0013] If the syntax element mixed_nalu_types_in_pic_flag is equal to 0, the value of the syntax element nal_unit_type is assumed to be the same for all coded slice NAL units of the picture. A picture or PU is referred to as having the same NAL unit type as the coded slice NAL units of the picture or PU.

[0014] Otherwise (where the syntax element mixed_nalu_types_in_pic_flag is equal to 1), one or more VCL NAL units shall all have a specific value of nal_unit_type within the range of IDR_W_RADL to CRA_NUT, including both ends, and all other VCL NAL units shall have a specific value of nal_unit_type within the range of TRAIL_NUT to RSV_VCL_6, including both ends, or be equal to GDR_NUT. [Overview of the Initiative]

[0015] The current design of the mixed VCL NAL unit type, as described in the background section above, may have several problems.

[0016] In some cases, the picture type of a picture may become ambiguous when the picture consists of mixed VCL NAL unit types.

[0017] In some cases, if NAL unit types are mixed within the same PU (Picture), temporal identifier (e.g., TemporalId) constraints may conflict.

[0018] For example, the current VVC specification has the following constraints regarding TemporalId: If the syntax element nal_unit_type is within the range of IDR_W_RADL to RSV_IRAP_12 (including both ends), the syntax element TemporalId shall be equal to 0. If the syntax element nal_unit_type is equal to STSA_NUT, the syntax element TemporalId shall not be equal to 0.

[0019] In some cases, if the syntax element mixed_nalu_types_in_pic_flag is signaled in the PPS, then at least two PPS NAL units shall be referenced by the slice NAL units in the CLVS. Also, if a subpicture is extracted, the associated PPS shall be rewritten by changing the value of the syntax element mixed_nalu_types_in_pic_flag.

[0020] In some cases, the current design may not support the coexistence of Random Access Decryptable Reading (RADL) / Random Access Skipped Reading (RASL) NAL units with Trail Pictures (PU) within a picture.

[0021] In some cases, when a picture in a layer references another picture in a different layer, the syntax element mixed_nalu_types_in_pic_flag may be inconsistent.

[0022] Embodiments of the present disclosure are capable of addressing one or more of the above-described problems and / or other problems.

[0023] According to one or more embodiments, a method is provided that is executed by at least one processor. The method includes: receiving a first video coding layer (VCL) network abstraction layer (NAL) unit of a first slice of an encoded picture and a second VCL NAL unit of a second slice of the encoded picture, wherein the first VCL NAL unit has a first VCL NAL unit type and the second VCL NAL unit has a second VCL NAL unit type that is different from the first VCL NAL unit type; and decoding the encoded picture, including determining a picture type of the encoded picture based on the first VCL NAL unit type of the first VCL NAL unit and the second VCL NAL unit type of the second VCL NAL unit, or based on an indicator received by at least one processor, the indicator indicating that the encoded picture includes different VCL NAL unit types.

[0024] According to an embodiment, the determining step includes determining that the encoded picture is a trailing picture based on the first VCL NAL unit including a trailing picture coding slice as indicated by the first VCL NAL unit type, and the second VCL NAL unit including an instant decoding refresh (IDR) picture coding slice or a clean random access (CRA) picture coding slice as indicated by the second VCL NAL unit type.

[0025] According to an embodiment, the determining step includes determining, based on that the coded picture is a Random Access Decodable Leading (RADL) picture, that the first VCL NAL unit includes a RADL picture coding slice, what the first VCL NAL unit type indicates, and that the second VCL NAL unit includes an Instantaneous Decodable Refresh (IDR) picture coding slice or a Clean Random Access (CRA) picture coding slice, what the second VCL NAL unit type indicates.

[0026] According to an embodiment, the determining step includes determining, based on that the coded picture is a step-wise temporal sub-layer access (STSA) picture, that the first VCL NAL unit includes a STSA picture coding slice, what the first VCL NAL unit type indicates, that the second VCL NAL unit does not include an Instantaneous Decodable Refresh (IDR) picture coding slice, and what the second VCL NAL unit type indicates.

[0027] According to an embodiment, the determining step includes determining, based on that the coded picture is a trailing picture, that the first VCL NAL unit includes a step-wise temporal sub-layer access (STSA) picture coding slice, what the first VCL NAL unit type indicates, and that the second VCL NAL unit does not include a Clean Random Access (CRA) picture coding slice, what the second VCL NAL unit type indicates.

[0028] According to the embodiment, the determination step includes determining that the coded picture is a trailing picture based on the first VCL NAL unit type indicating that the first VCL NAL unit includes a gradual decoding refresh (GDR) picture coding slice, and that the second VCL NAL unit type indicating that the second VCL NAL unit does not include an immediate decoding refresh (IDR) picture coding slice or a clean random access (CRA) picture coding slice.

[0029] According to one embodiment, the indicator is a flag, and the determining step includes determining that the coded picture is a trailing picture based on the flag indicating that the coded picture contains mixed VCL NAL unit types.

[0030] According to one embodiment, the indicator is a flag, and the step of decoding the coded picture further includes determining that the temporal ID of the coded picture is 0 based on the flag indicating that the coded picture contains a mixed VCL NAL unit type.

[0031] According to one embodiment, the indicator is a flag, and the method further includes the step of receiving the flag in a picture header or slice header.

[0032] According to one embodiment, the indicator is a flag, the coded picture is in a first layer, and the method further includes the steps of: receiving the flag; determining, based on the flag indicating, that an additional coded picture in a second layer, which is a reference layer of the first layer, includes a mixed VCL NAL unit type.

[0033] According to one or more embodiments, a system is provided. The system includes: a memory configured to store a computer program; and at least one processor configured to receive at least one coded video stream, access computer program code, and operate as directed by the computer code. The computer program code includes a decoding code configured to cause at least one processor to decode a coded picture from at least one coded video stream, the decoding code including a determination code configured to cause at least one processor to determine the picture type of the coded picture based on a first VCL NAL unit type of a first video coding layer (VCL) network abstraction layer (NAL) unit of a first slice of the coded picture and a second VCL NAL unit type of a second VCL NAL unit of a second slice of the coded picture, or based on an indicator received by at least one processor indicating that the coded picture contains a mixed VCL NAL unit type, wherein the first VCL NAL unit type is different from the second VCL NAL unit type.

[0034] According to one embodiment, the decision code is configured to cause at least one processor to determine that the coded picture is a trailing picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit includes a trailing picture coded slice, and the second VCL NAL unit type indicating that the second VCL NAL unit includes an immediate decoded refresh (IDR) picture coded slice or a clean random access (CRA) picture coded slice.

[0035] According to one embodiment, the decision code is configured to cause at least one processor to determine that the coded picture is a random access decodeable reading (RADL) picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit includes a RADL picture coded slice, and the second VCL NAL unit type indicating that the second VCL NAL unit includes an immediate decodeable refresh (IDR) picture coded slice or a clean random access (CRA) picture coded slice.

[0036] According to one embodiment, the decision code is configured to cause at least one processor to determine that the coded picture is a stepwise temporal sublayer access (STSA) picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit includes an STSA picture coded slice, and the second VCL NAL unit type indicating that the second VCL NAL unit does not include an immediate decoded refresh (IDR) picture coded slice.

[0037] According to one embodiment, the decision code is configured to cause at least one processor to determine that the coded picture is a trailing picture based on the first VCL NAL unit type indicating that the first VCL NAL unit includes a stepwise temporal sublayer access (STSA) picture coded slice, and the second VCL NAL unit type indicating that the second VCL NAL unit does not include a clean random access (CRA) picture coded slice.

[0038] According to one embodiment, the decision code is configured to cause at least one processor to determine that the coded picture is a trailing picture based on the first VCL NAL unit type indicating that the first VCL NAL unit includes a progressive decoded refresh (GDR) picture coded slice, and the second VCL NAL unit type indicating that the second VCL NAL unit does not include an immediate decoded refresh (IDR) picture coded slice or a clean random access (CRA) picture coded slice.

[0039] According to one embodiment, the indicator is a flag, and the decision code is configured to cause at least one processor to determine that the coded picture is a trailing picture based on the flag indicating that the coded picture includes a mixed VCL NAL unit type.

[0040] According to one embodiment, the indicator is a flag, and the determination code is further configured to cause at least one processor to determine that the temporal ID of the coded picture is 0, based on the flag indicating that the coded picture contains a mixed VCL NAL unit type.

[0041] According to one embodiment, the indicator is a flag, and at least one processor is configured to receive the flag in a picture header or slice header.

[0042] According to one or more embodiments, a non-temporary computer-readable medium for storing computer instructions is provided. The computer instructions, when executed by at least one processor, cause at least one processor to decode a coded picture from at least one coded video stream, the decoding including determining the picture type of the coded picture based on a first VCL NAL unit type of a first video coding layer (VCL) network abstraction layer (NAL) unit of a first slice of the coded picture and a second VCL NAL unit type of a second VCL NAL unit of a second slice of the coded picture, or based on an indicator received by at least one processor indicating that the coded picture includes a mixed VCL NAL unit type, wherein the first VCL NAL unit type is different from the second VCL NAL unit type. [Brief explanation of the drawing]

[0043] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings.

[0044] [Figure 1] This is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0045] [Figure 2] This is a schematic diagram of a simplified block diagram of a communication system according to an embodiment.

[0046] [Figure 3] This is a schematic diagram of a simplified block diagram of a decoder according to an embodiment.

[0047] [Figure 4] This is a schematic diagram of a simplified block diagram of an encoder according to an embodiment.

[0048] [Figure 5] This is a block diagram of the NAL unit according to an embodiment.

[0049] [Figure 6] This is a block diagram of a decoder according to an embodiment.

[0050] [Figure 7] This is a diagram of a computer system suitable for carrying out an embodiment. [Modes for carrying out the invention]

[0051] Figure 1 shows a simplified block diagram of a communication system (100) according to an embodiment of the present disclosure. The system (100) may include at least two terminals (110, 120) interconnected via a network (150). For one-way data transmission, the first terminal (110) can encode video data at its local location for transmission to the other terminal (120) via the network (150). The second terminal (120) can receive the encoded video data from the other terminal via the network (150), decode the encoded data, and display the restored video data. One-way data transmission is common in media serving applications and the like.

[0052] Figure 1 shows a second pair of terminals (130, 140) provided to support the bidirectional transmission of coded video, for example, during a video conference. For bidirectional data transmission, each terminal (130, 140) can code video data captured at its local location for transmission to the other terminal over the network (150). Each terminal (130, 140) can also receive coded video data transmitted by the other terminal, decode the coded data, and display the restored video data on a local display device.

[0053] In Figure 1, terminals (110-140) may be represented as servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (150) represents any number of networks that transmit coded video data between terminals (110-140), including, for example, wired and / or wireless communication networks. Communication network (150) can exchange data over circuit-switched and / or packet-switched channels. Typical networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network (150) may not be important to the operation of this disclosure unless described below.

[0054] Figure 2 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and decoder in a streaming environment. The disclosed subject matter can also be similarly applied to other video-enabled applications, including, for example, video conferencing, digital TV, and storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.).

[0055] As shown in Figure 2, the streaming system (200) may include a capture system (213) which may include a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera and may be configured to generate an uncompressed video sample stream (202). The uncompressed video sample stream (202) provides a higher data volume compared to an encoded video bitstream and can be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination thereof and can enable or realize aspects of the disclosed subject matter as described in detail below. The encoded video bitstream (204) may contain a lower data volume compared to the sample stream and can be stored in a streaming server (205) for future use. One or more streaming clients (206) can access a streaming server (205) to search for a video bitstream (209) which may be a copy (207) of an encoded video bitstream (204).

[0056] In an embodiment, the streaming server (205) may function as a media-aware network element (MANE). For example, the streaming server (205) can be configured to prune an encoded video bitstream (204) to align potentially different bitstreams for one or more streaming clients (206). In an embodiment, the MANE may be provided separately from the streaming server (205) within the streaming system (200).

[0057] A streaming client (206) may include a video decoder (210) and a display (212). The video decoder (210) can, for example, decode a video bitstream (209), which is an incoming copy of an encoded video bitstream (204), and generate a progressing video sample stream (211) that can be rendered on a display (212) or other rendering device (not shown). In some streaming systems, the video bitstreams (204, 209) may be encoded according to a specific video coding / compression standard. Specific examples of these standards include, but are not limited to, ITU-T Recommendation H.265. A video coding standard informally known as Versatile Video Coding (VVC) is under development. Embodiments of the disclosure may be used in the context of VVC.

[0058] Figure 3 shows an exemplary functional block diagram of a video decoder (210) attached to a display (212) according to an embodiment of the present disclosure.

[0059] The video decoder (210) may include channels (312), a receiver (310), buffer memory (315), an entropy decoder / parser (320), a scaler / inverse unit (351), an intra-prediction unit (352), a motion compensation prediction unit (353), an aggregator (355), a loop filter unit (356), a reference picture memory (357), and a memory for the current picture (357). In at least one embodiment, the video decoder (210) may include an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The video decoder (210) may also be implemented in software running on one or more CPUs, together with the associated memory, in part or in whole.

[0060] In this embodiment and other embodiments, the receiver (310) can receive one or more coded video sequences to be decoded by the decoder (210) in one coded video sequence at a time, and the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences can be received from a channel (312), which may be a hardware / software link to a storage device that stores coded video data. The receiver (310) can receive coded video data together with other data, such as coded audio data and / or auxiliary data streams, which can be transmitted using entities (not shown). The receiver (310) can isolate coded video sequences from other data. To address network jitter, a buffer memory (315) may be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as the “Parser”). If the receiver (310) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer (315) may be unnecessary or small. For use in best-effort packet networks such as the Internet, the buffer (315) may be required and can be relatively large and adaptively sized.

[0061] The video decoder (210) may include a parser (320) for reconstructing symbols (321) from an entropy-encoded video sequence. These symbols may include, for example, information used to manage the operation of the decoder (210) and information that may control rendering devices, such as a display (212) that can be coupled to the decoder, as shown in Figure 2. The rendering device control information may be in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (320) can parse / entropy-decode the coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context influence. The parser(320) can extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to a group. Subgroups can include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), and the like. The parser(320) can also extract information such as transform coefficients, quantization parameter values, and motion vectors from the coded video sequence.

[0062] The parser (320) can perform an entropy decoding / analysis operation on the video sequence received from the buffer (315) in order to create a symbol (321).

[0063] The reconstruction of the symbol (321) may involve multiple different units depending on the type of coded video picture or part thereof (e.g., between pictures, within a picture, between blocks, within a block) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information analyzed by the parser (320) from the coded video sequence. The flow of such subgroup control information between the parser (320) and the subsequent multiple units is not depicted for clarity.

[0064] Beyond the functional blocks already mentioned, the decoder 210 can conceptually be subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and be at least partially integrated. However, the following conceptual subdivision into functional units is appropriate to illustrate the subject matter being disclosed.

[0065] One unit may be a scaler / inverse unit (351). The scaler / inverse unit (351) receives quantized transformation coefficients as well as control information, which includes symbols (321) from the parser (320) indicating which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc. The scaler / inverse unit (351) can output a block containing sample values ​​that can be input to the aggregator (355).

[0066] In some cases, the output samples of the scaler / inverse transform (351) can be associated with intra-coded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information can be provided by the intra-picture predictive unit (352). In some cases, the intra-picture predictive unit (352) generates blocks of the same size and shape as the block being reconstructed, using already reconstructed information from the surrounding area taken from the current (partially reconstructed) picture in the current picture memory (358). The aggregator (355) adds the predictive information generated by the intra-predictive unit (352) to the output sample information, as provided by the scaler / inverse transform unit (351), possibly sample by sample.

[0067] In other cases, the output samples of the scaler / inverse unit (351) can be associated with an intercoded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (353) can access the reference picture memory (357) to retrieve samples to be used for prediction. After motion-compensating the retrieved samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler / inverse unit (in this case, called residual samples or residual signals) to generate output sample information. The address in the reference picture memory (357) from which the motion-compensated unit retrieves prediction samples can be controlled by a motion vector. The motion vector may be available to the motion-compensated prediction unit (353) in the form of a symbol (321) which may have, for example, X, Y, and reference picture components. Furthermore, motion compensation may include interpolation of sample values, motion vector prediction mechanisms, etc., when precise motion vectors of subsamples are used, such as those retrieved from a reference picture memory (357).

[0068] The output samples of the aggregator (355) can be affected by various loop filtering techniques within the loop filtering unit (356). The video compression technique may include an in-loop filtering technique, which is controlled by parameters contained in the coded video bitstream and made available to the loop filtering unit (356) as symbols (321) from the parser (320), but can also respond to metadata obtained during the decoding of earlier parts (in decoding order) of the coded picture or coded video sequence, and can also respond to previously reconstructed and loop-filtered sample values.

[0069] The output of the loop filter unit (356) can be a sample stream that can be output to a rendering device such as a display (212) and can also be stored in reference picture memory for use in future inter-picture prediction.

[0070] A particular coded picture, once fully reconstructed, can be used as a reference picture for future predictions. Once a coded picture is fully reconstructed and identified as a reference picture (for example, by the parser (320)), the current reference picture can become part of the reference picture memory (357), and new memory for the current picture can be reallocated before any subsequent coding picture reconstructions begin.

[0071] The video decoder (210) can perform decoding operations according to a given video compression technique, which may be documented in a standard such as ITU-T Rec.H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to the syntax of the video compression technique or standard, as specified in the video compression technique documentation or standard, particularly in the profile documentation within it. Also, for compliance with certain video compression techniques or standards, the complexity of the coded video sequence may be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limitations set by the level may, in some cases, be further limited by metadata for HRD buffer management and Hypothetical Reference Decoder (HRD) specifications signaled in the coded video sequence.

[0072] In the embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be, for example, in the form of temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0073] Figure 4 shows an exemplary functional block diagram of a video encoder (203) associated with a video source (201) according to an embodiment of the present disclosure.

[0074] The video encoder (203) may include, for example, an encoder comprising a source coder (430), a coding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter (440), an entropy coder (445), a controller (450), and channels (460).

[0075] The encoder (203) can receive video samples from a video source (201) (not part of the encoder) that is capable of capturing video images to be coded by the encoder (203).

[0076] The video source (201) can provide a source video sequence coded by the encoder (203) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (201) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (203) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following explanation focuses on the sample.

[0077] According to the embodiment, the encoder (203) can encode and compress the pictures of a source video sequence into a coded video sequence (443) in real time or under any other temporal constraints required by the application. Enforcing an appropriate coding speed is one function of the controller (450). The controller can also control other functional units as described below and can be functionally coupled to these units. The coupling is not depicted for clarity. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will be able to easily identify other functions of the controller (450) because they may relate to a video encoder (203) optimized for a particular system design.

[0078] Some video encoders operate in what is readily recognizable to those skilled in the art as a "coding loop." In an extremely simplified explanation, the coding loop may consist of an encoding portion comprising a source coder (430) (responsible for generating symbols based on the input and reference pictures to be coded) and a (local) decoder (433) incorporated into the encoder (203), the (local) decoder (433) reconstructing the symbols to create sample data, and a (remote) decoder will also create its sample data if the compression between the symbols and the coded video bitstream is lossless in the video compression technique. The reconstructed sample stream can be input to a reference picture memory (434). Since the decoding of the symbol stream yields a bit-exact result independent of the decoder's location (local or remote), the contents of the reference picture memory are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder would "see" if it were using the prediction during decoding. This fundamental principle of reference picture synchronization (for example, if synchronization cannot be maintained due to channel errors, drift will result) is well known to those skilled in the art.

[0079] The operation of the “local” decoder (433) can be the same as that of the “remote” decoder (210), which has already been described in detail in relation to Figure 3. However, since symbols are available and the encoding / decoding of symbols by the entropy coder (445) and parser (320) to the coded video sequence can be lossless, the entropy decoding portion of the decoder (210), including the channel (312), receiver (310), buffer (315), and parser (320), does not need to be fully implemented in the local decoder (433).

[0080] An insight that can be gained at this point is that any decoder technique present in the decoder, excluding analysis / entropy decoding, must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. A description of encoder techniques can be omitted, as it may be the inverse of a comprehensive description of decoder techniques. More detailed explanations are required only in specific areas and are provided below.

[0081] The source coder (430) can, as part of its operation, perform motion-compensated predictive coding, predictively coding an input frame while referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this way, the coding engine (432) codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame which may be selected as a predictive reference for the input frame.

[0082] The local video decoder (433) can decode the encoded video data of a frame that may be designated as a reference frame, based on the symbols generated by the source coder (430). The operation of the coding engine (432) may, advantageously, be a lossless process. If the encoded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (433) can replicate the decoding process that may be performed by the video decoder on the reference frame, causing the reconstructed reference frame to be stored in the reference picture memory (434). In this way, the encoder (203) can store a locally reconstructed copy of the reference frame that has common content with the reconstructed reference frame obtained by the video decoder at the far end (assuming no transmission errors).

[0083] The predictor (435) can perform a predictive search for the coding engine (432). That is, for a new frame to be coded, the predictor (435) can search the reference picture memory (434) for sample data (as candidate reference pixel blocks) or specific metadata (e.g., reference picture motion vectors, block shapes, etc.), which may serve as appropriate predictive references for the new picture. The predictor (435) can operate block by block on the samples to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (434), as determined by the search results obtained by the predictor (435).

[0084] The controller (450) can manage the coding operations of the video coder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0085] The outputs of all the aforementioned functional units may be subject to entropy coding in the entropy coder (445). The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0086] The transmitter (440) can buffer coded video sequences, such as those created by the entropy coder (445), and prepare them for transmission over a communication channel (460), which may be a hardware / software link to a storage device that will store the coded video data. The transmitter (440) can merge the coded video data from the video coder (430) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0087] The controller (450) can manage the operation of the encoder (203). During coding, the controller (450) can assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to individual pictures. For example, pictures may often be designated as intra-pictures (I-pictures), predictive pictures (P-pictures), or bidirectional predictive pictures (B-pictures).

[0088] An intra-picture (I-picture) may be one that can be encoded and decoded without using any other frames in the sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoded Refresh (IDR) pictures. Those skilled in the art are familiar with these variations of I-pictures, as well as their respective uses and characteristics.

[0089] A predictive picture (P-picture) may be encoded and decoded using intra-prediction or inter-prediction, with up to one motion vector and a reference index to predict the sample values ​​of each block.

[0090] A bidirectional predictive picture (B-picture) may be encoded and decoded using intra-prediction or inter-prediction, with up to two motion vectors and a reference index to predict the sample values ​​of each block. Similarly, multiple predictive pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0091] A source picture can typically be spatially divided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks can be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded unpredictably, or they may be predictively coded (spatially predictively or intrapredictively) by referencing already coded blocks of the same picture. Pixel blocks of picture P may be coded unpredictably by spatial or temporal prediction, referencing one previously coded reference picture. Blocks of picture B may be coded unpredictably by spatial or temporal prediction, referencing one or two previously coded reference pictures.

[0092] The video coder (203) can perform coding operations in accordance with a specified video coding technique or standard, such as ITU-T Rec.H.265. During operation, the video coder (203) can perform various compression operations, including predictive coding operations that leverage temporal and spatial redundancy in the input video sequence. Therefore, the coded video data can conform to the syntax specified by the video coding technique or standard being used.

[0093] In the embodiment, the transmitter (440) may transmit additional data along with the encoded video. The video coder (430) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0094] Embodiments of this disclosure may modify the current VVC specification and may implement the NAL unit type codes and NAL unit type classes defined in Table 2 above.

[0095] An "Intra-Random Access Point Picture" (or "IRAP Picture") can be a picture that does not reference any other pictures for interpretation in the decoding process, and may be a Clean Random Access Picture (CRA) or an Immediate Decoded Refresh (IDR) picture. The first picture in the bitstream in the decoding order may be an IRAP or a Gradual Decoded Refresh (GDR) picture. If the necessary set of parameters is available where they need to be referenced, the IRAP picture and all subsequent non-RASL pictures in the coded video sequence (CVS) in the decoding order can be correctly decoded without performing the decoding process of any picture preceding the IRAP picture in the decoding order.

[0096] A "trailing picture" is a non-IRAP picture that follows the related IRAP picture in the output order, and may not be a Stepwise Temporal Sublayer Access (STSA) picture.

[0097] A "Stepwise Temporal Sublayer Access Picture" (or "STSA Picture") may be a picture that does not use a picture with the same TemporalId as the STSA Picture for interpredictive referencing. A picture that follows the STSA Picture in decoding order and has the same TemporalId as the STSA Picture may not use a picture that precedes the STSA Picture in decoding order and has the same TemporalId as the STSA Picture for interpredictive referencing. An STSA Picture may allow up-switching from the sublayer directly below it to the sublayer containing the STSA Picture. An STSA Picture may have a TemporalId greater than 0.

[0098] A "Random Access Skip Reading Picture" (or "RASL Picture") may be the reading picture of the associated CRA Picture. If the associated CRA Picture has a NoIncorrectPicOutputFlag equal to 1, the RASL Picture may not be output and may not be decoded correctly because the RASL Picture may contain references to pictures that do not exist in the bitstream. The RASL Picture may not be used as a reference picture in the decoding process of non-RASL Pictures. If present, all RASL Pictures may precede all trailing pictures of the same associated CRA Picture in decoding order.

[0099] A “Random Access Decryptable Reading Picture” (or “RADL Picture”) may be a reading picture that is not used as a reference picture for the decoding process of the trailing picture of the same associated IRAP picture. If present, all RADL pictures may precede all trailing pictures of the same associated IRAP picture in decoding order.

[0100] An "immediate decryption refresh picture" (or "IDR picture") may be a picture that does not have an associated reading picture present in the bitstream (for example, nal_unit_type is equal to IDR_N_LP), or a picture that does not have an associated RASL picture present in the bitstream but may have an associated RADL picture in the bitstream (for example, nal_unit_type is equal to IDR_W_RADL).

[0101] A "clean random access picture" (or "CRA picture") may be a picture that, in its decoding process, does not reference any other pictures for interpretation, and may be the first picture in the bitstream in the decoding order, or it may appear later in the bitstream. A CRA picture may have associated RADL or RASL pictures. If a CRA picture has a NoIncorrectPicOutputFlag equal to 1, associated RASL pictures may not be output by the decoder because they may not be decodeable, since they may contain references to pictures that are not present in the bitstream.

[0102] According to one or more embodiments, if the syntax element mixed_nalu_types_in_pic_flag of the PPS referenced by the encoded picture is equal to 1, the picture type of the encoded picture is determined (for example, by the decoder) as follows:

[0103] (A) If the nal_unit_type of the picture's NAL unit is equal to TRAIL_NUT, and the nal_unit_type of the other NAL units of the picture is within the range of IDR_W_RADL to CRA_NUT, then the picture is determined to be a trailing picture.

[0104] (B) If the nal_unit_type of the picture's NAL unit is equal to RADL_NUT, and the nal_unit_type of the other NAL units of the picture is within the range of IDR_W_RADL or CRA_NUT, the picture is determined to be a RADL picture.

[0105] (C) If the nal_unit_type of the picture's NAL unit is equal to STSA_NUT, and the nal_unit_type of the other NAL units of the picture is IDR_W_RADL or IDR_N_LP, the picture is determined to be an STSA picture.

[0106] (D) If the nal_unit_type of the picture's NAL unit is equal to STSA_NUT and the nal_unit_type of the other NAL units of the picture is CRA_NUT, the picture is determined to be a trailing picture.

[0107] (E) If the nal_unit_type of the picture's NAL unit is equal to GDR_NUT, and the nal_unit_type of the other NAL units of the picture is within the range of IDR_W_RADL or CRA_NUT, the picture is determined to be a trailing picture.

[0108] According to one or more embodiments, if the syntax element mixed_nalu_types_in_pic_flag of the PPS referenced by the encoded picture is equal to 1, the picture type of the encoded picture is determined (e.g., by the decoder) to be a trailing picture.

[0109] The above embodiments can provide a solution to "Problem 1" as described in the section outlining the invention above.

[0110] According to one or more embodiments, mixing of STSA NAL units with IRAP NAL units may not be permissible.

[0111] For example, the following can be implemented for any picture's VCL NAL unit:

[0112] If mixed_nalu_types_in_pic_flag is equal to 0, the value of nal_unit_type is assumed to be the same for all coded slice NAL units in the picture (for example, it can be determined that they are the same). A picture or PU is referred to as having the same NAL unit type as the coded slice NAL units of the picture or PU.

[0113] Otherwise (where mixed_nalu_types_in_pic_flag is equal to 1), one or more VCL NAL units shall all have a specific value of nal_unit_type within the range of IDR_W_RADL to CRA_NUT, including both ends (it is possible to determine that all have this), and all other VCL NAL units shall have a specific value of nal_unit_type within the range of RADL_NUT to RSV_VCL_6, including both ends (it is possible to determine that all have this), or be equal to GDR_NUT or TRAIL_NUT.

[0114] According to one embodiment, the encoder may be configured to apply the above in order to prevent mixing of STSA NAL units with IRAP NAL units. According to another embodiment, the decoder may be configured to determine the value of the NAL unit type based on the above.

[0115] According to one or more embodiments, the TemporalId constraint for STSA_NUT in the current VVC specification draft JVET-P2001 can be removed.

[0116] In other words, embodiments of the present disclosure do not have to implement the constraint that, for example, if nal_unit_type is equal to STSA_NUT, then TemporalId is not equal to 0. However, embodiments may still implement the constraint that, if nal_unit_type is within the range of IDR_W_RADL to RSV_IRAP_12, including both ends, then TemporalId is equal to 0 (for example, it can be determined to be equal).

[0117] According to one or more embodiments, a constraint may be implemented that the TemporalId of a picture having a mixed_nalu_types_in_pic_flag equal to 1 is equal to 0. For example, an encoder or decoder of the present disclosure may determine that the TemporalId of a picture is 0 based on the fact that the flag mixed_nalu_types_in_pic_flag is equal to 1.

[0118] The above embodiments can provide a solution to "Problem 2" described in the section outlining the invention above.

[0119] According to one or more embodiments, the syntax element mixed_nalu_types_in_pic_flag may be provided in a picture header or slice header instead of a PPS. An example of the syntax element mixed_nalu_types_in_pic_flag in a picture header is provided in Table 3 below. Table 3 [Table 3]

[0120] A syntax element mixed_nalu_types_in_pic_flag equal to 1 can specify that each picture associated with a PH has more than one VCL NAL unit, that no VCL NAL units have the same nal_unit_type, and that the pictures are not IRAP pictures. A syntax element mixed_nalu_types_in_pic_flag equal to 0 can specify that each picture associated with a PH has one or more VCL NAL units, and that the VCL NAL units of each picture associated with a PH have the same nal_unit_type.

[0121] If the syntax element no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, then the value of mixed_nalu_types_in_pic_flag is equal to 0 (for example, this can be determined).

[0122] According to one or more embodiments, the syntax element mixed_nalu_types_in_pic_flag may be provided in the picture header or slice header along with the current flag in the SPS.

[0123] Examples of SPS with the current flag (sps_mixed_nalu_types_present_flag) are provided in Table 4 below. Table 4 [Table 4]

[0124] An example of a picture header with the syntax element mixed_nalu_types_in_pic_flag is provided in Table 5 below. Table 5 [Table 5]

[0125] A syntax element sps_mixed_nalu_types_present_flag equal to 1 can specify that zero or more images referencing an SPS have multiple VCL NAL units, that the VCL NAL units do not have the same nal_unit_type, and that the picture is not an IRAP picture. A syntax element sps_mixed_nalu_types_present_flag equal to 0 can specify that each picture referencing an SPS has one or more VCL NAL units, and that each picture referencing a PPS has the same nal_unit_type.

[0126] If the syntax element no_mixed_nalu_types_in_pic_constraint_flag is equal to 1, then the value of the syntax element sps_mixed_nalu_types_present_flag is equal to 0 (for example, this can be determined).

[0127] A syntax element mixed_nalu_types_in_pic_flag equal to 1 can specify that each picture associated with a PH has multiple VCL NAL units, that the VCL NAL units do not have the same nal_unit_type, and that the picture is not an IRAP image. A syntax element mixed_nalu_types_in_pic_flag equal to 0 can specify that each picture associated with a PH has one or more VCL NAL units, and that the VCL NAL units of each picture associated with a PH have the same nal_unit_type. If it does not exist, the value of mixed_nalu_types_in_pic_flag may be inferred to be equal to 0 (for example, by the decoder).

[0128] The above embodiments can provide a solution to "Problem 3" described in the section outlining the invention above.

[0129] According to one or more embodiments, the syntax element flag mixed_nalu_types_in_pic_flag can be replaced with the indicator mixed_nalu_types_in_pic_idc.

[0130] An example of a picture parameter set with the syntax element mixed_nalu_types_in_pic_idc is provided in Table 6 below. Table 6 [Table 6]

[0131] A syntax element mixed_nalu_types_in_pic_idc equal to 1 or 2 can specify that each picture referencing the PPS has multiple VCL NAL units, that the VCL NAL units do not have the same nal_unit_type, and that the picture is not an IRAP image. A syntax element mixed_nalu_types_in_pic_idc equal to 0 can specify that each picture referencing the PPS has one or more VCL NAL units, and that the VCL NAL units of each picture referencing the PPS have the same nal_unit_type. Other values ​​for the syntax element mixed_nalu_types_in_pic_idc may be reserved for future use by ITU-T|ISO / IEC.

[0132] If the syntax element no_mixed_nalu_types_in_pic_constraint_idc is equal to 1, then the value of mixed_nalu_types_in_pic_idc is assumed to be equal to 0 (which can be determined, for example, by the decoder).

[0133] For each slice where the value of nal_unit_type nalUnitTypeA is within the range of IDR_W_RADL or CRA_NUT including both ends, in picture picA which also includes one or more slices with a different value of nal_unit_type (i.e., the value of mixed_nalu_types_in_pic_idc in picture picA is equal to 1), the following can be implemented:

[0134] (A) A slice is considered to belong to subpicA whose corresponding subpic_treated_as_pic_flag[i] value is equal to 1 (for example, it can be determined that it belongs to it).

[0135] (B) A slice shall not belong to any subpicture of picA that contains a VCL NAL unit having a nal_unit_type that is not equal to nalUnitTypeA (for example, it may be determined that it does not belong).

[0136] (C) With respect to all subsequent PUs in the CLVS in the decoding order, neither RefPicList[0] nor RefPicList[1] of the slice in subpicA shall contain any picture that precedes picA in the decoding order in the active entry.

[0137] RefPicList[0] may be a list of reference pictures used for interpretation of P slices, or a first list of reference pictures used for interpretation of B slices. RefPicList[1] may be a second list of reference pictures used for interpretation of B slices.

[0138] For any specific picture's VCL NAL unit, the following can be implemented:

[0139] (A) If the syntax element mixed_nalu_types_in_pic_idc is equal to 1, then all one or more VCL NAL units shall have a specific value of nal_unit_type within the range of IDR_W_RADL to CRA_NUT, including both ends (for example, it may be determined that they do), and all other VCL NAL units shall have a specific value of nal_unit_type within the range of TRAIL_NUT to RSV_VCL_6, including both ends (for example, it may be determined that they do), or be equal to GDR_NUT.

[0140] (B) If the syntax element mixed_nalu_types_in_pic_idc is equal to 2, then all one or more VCL NAL units have a specific value of nal_unit_type that is equal to RASL_NUT or RADL_NUT including both ends (for example, it can be determined that they do), or equal to GDR_NUT, and all other VCL NAL units have a specific value of nal_unit_type within the range of TRAIL_NUT to RSV_VCL_6 including both ends (for example, it can be determined that they do), or equal to GDR_NUT, and their nal_unit_types are different from the other nal_unit_types.

[0141] (C) Otherwise (mixed_nalu_types_in_pic_idc is equal to 0), the value of nal_unit_type shall be the same for all coded slice NAL units of the picture (for example, it may be determined that they are the same). A picture or PU is referred to as having the same NAL unit type as the coded slice NAL units of the picture or PU.

[0142] If the mixed_nalu_types_in_pic_idc of the PPS referenced in the coded picture is equal to 1 or 2, the picture is determined to be a trailing picture (e.g., by the decoder).

[0143] The above embodiments can provide a solution to "Problem 4" described in the section outlining the invention above.

[0144] According to one or more embodiments, if the syntax element mixed_nalu_types_in_pic_flag of a picture in layer A is equal to 1, then the mixed_nalu_types_in_pic_flag of a picture in layer B, which is the reference layer of layer A, is assumed to be equal to 1 in the same AU (for example, it is possible to determine that they are equal).

[0145] The above embodiments can provide a solution to "Problem 5" described in the section outlining the invention above.

[0146] According to one or more embodiments, one or more coded video data bitstreams, as well as the syntax structure and elements therein (such as the VCL NAL units and parameter sets described above), may be received by a decoder of the Disclosure for decoding the received video data. The decoder of the Disclosure can decode a coded picture of video based on VCL NAL units of a coded picture having mixed VCL NAL unit types (e.g., VCL NAL unit (500) shown in Figure 5) according to embodiments of the Disclosure.

[0147] For example, referring to Figure 6, the decoder (600) may include a decoding code (610) configured to cause at least one processor of the decoder (600) to decode an encoded picture based on a VCL NAL unit. According to one or more embodiments, the decoding code (610) may include a determination code (620) which, as described in embodiments of the present disclosure, (a) determines or restricts the NAL unit type of one or more VCL NAL units of the coded picture based on the NAL unit type of another one or more VCL NAL units of the coded picture, or based on an indicator (e.g., a flag); (b) determines or restricts the picture type of the coded picture based on one or more NAL unit types of the VCL NAL units of the coded picture, or based on an indicator (e.g., a flag); (c) determines or restricts the TemporalID of the coded picture based on one or more VCL NAL unit types of one or more VCL NAL units of the coded picture, or based on an indicator (e.g., a flag); and / or (d) the coded picture has multiple VCLs having mixed VCL NAL unit types. The decoder (600) is configured to cause at least one processor to determine or restrict an indicator (e.g., a flag) indicating whether or not it has a NAL unit, based on another indicator (e.g., a flag) that it has received or determined.

[0148] Embodiments of the present disclosure can be used separately or in any combination of any order. Furthermore, each of the methods, encoders, and decoders of the present disclosure may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium.

[0149] The technology described above is computer software that uses computer-readable instructions and can be implemented as being physically stored on one or more computer-readable media. For example, Figure 7 shows a computer system (900) suitable for carrying out embodiments of the disclosed subject matter.

[0150] Computer software can be coded using any suitable machine code or computer language that is subject to mechanisms such as assembly, compilation, and linking in order to generate code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by a computer's central processing unit (CPU), graphics processing unit (GPU), etc.

[0151] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things (IoT) devices.

[0152] The components shown in Figure 7 relating to the computer system (900) are essentially illustrative and are not intended to imply any limitation on the scope of use or functionality of the computer software that realizes embodiments of the present disclosure. Furthermore, the configuration of the components should not be construed as having any dependency or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (900).

[0153] The computer system (900) may include certain human interface input devices. Such human interface input devices can respond to input from one or more human users, for example, via haptic input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, applause), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., conversations, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), or video (e.g., 2D video, 3D video including stereoscopic video).

[0154] An input human interface device may include one or more of the following (although only one of each is depicted): a keyboard (901), a mouse (902), a trackpad (903), a touchscreen (910), a data glove, a joystick (905), a microphone (906), a scanner (907), and a camera (908).

[0155] The computer system (900) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (910), data glove, or joystick (905), although there may also be tactile feedback devices that do not function as input devices). For example, such devices may include audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (screens (910), including CRT screens, LCD screens, plasma screens, OLED screens, each having or not having touchscreen input capabilities, each having or not having haptic feedback capabilities, some of which are capable of outputting three or more dimensions by means such as two-dimensional visual output or stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0156] The computer system (900) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (920) having CD / DVD or similar media (921), thumb drives (922), removable hard drives or solid-state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0157] Those skilled in the art will also understand that the term “computer-readable medium” as used in connection with the subject matter of this disclosure does not include a transmission medium, carrier wave, or other transient signal.

[0158] The computer system (900) may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. Furthermore, these networks may be local, wide-area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include Ethernet, wireless LAN, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), wired or wireless wide-area digital TV networks (including cable TV, satellite TV, and terrestrial TV), and vehicle and industrial networks including CANBus. Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system (900)) connected to a specific general-purpose data port or peripheral bus (949); other networks are generally integrated into the core of the computer system (900) by connection to a system bus, as described below (e.g., an Ethernet interface is integrated into a PC computer system, and a cellular network interface is integrated into a smartphone computer system). Using any of these networks, a computer system (900) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Such communication can include communication to a cloud computing environment (955). Specific protocols and protocol stacks can be used for each of these networks and network interfaces, as described above.

[0159] The aforementioned human interface device, human-accessible storage device, and network interface (954) can be mounted on the core (940) of the computer system (900).

[0160] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specific programmable processing units in the form of field-programmable gate areas (FPGAs) (943), hardware accelerators (944) for specific tasks, and so on. These devices may be connected via a system bus (948) along with read-only memory (ROM) (945), random-access memory (RAM) (946), and internal mass storage, such as internal user-inaccessible hard drives, SSDs, and similar devices (947). In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (948) or via a peripheral bus (949). The architecture of the peripheral bus may include PCI, USB, etc. The graphics adapter 950 may be included in the core 940.

[0161] The CPU (941), GPU (942), FPGA (943), and accelerator (944) can be combined to execute specific instructions that can construct the aforementioned computer code. This computer code can be stored in ROM (945) or RAM (946). Temporary data can also be stored in RAM (946), while persistent data can be stored, for example, in internal mass storage (947). By using cache memory that can be closely associated with one or more CPUs (941), GPUs (942), mass storage (947), ROMs (945), RAM (946), etc., high-speed storage and retrieval of any memory device can be enabled.

[0162] Computer-readable media can contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art of computer software.

[0163] As an example, and not an limitation, a computer system having an architecture (900), specifically a core (940), can provide functions derived from a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage as described above, as well as specific storage of the core (940) of a non-transient nature, such as core internal mass storage (947) or ROM (945). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (940). The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core (940), in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to execute a specific process or a specific part of a specific process as described herein, including defining data structures stored in RAM (946) and modifying such data structures according to a process defined by the software. Furthermore, or alternatively, a computer system may provide functionality derived from logic wired within a circuit (e.g., an accelerator (944)) or otherwise incorporated therein, the circuit may operate in place of or with software to perform a particular process or a particular part of a particular process as described herein. References to software may include logic, and vice versa, as appropriate. References to computer-readable media may include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both, as appropriate. This disclosure encompasses any appropriate combination of hardware and software.

[0164] While this disclosure describes several exemplary, non-limiting embodiments, there are many modifications, substitutions, and alternative equivalents that fall within the scope of this disclosure. Therefore, it will be acknowledged that many systems and methods, not expressly illustrated or described in this application, that embody the principles of this disclosure and thus fall within its spirit and scope, can be devised by those skilled in the art.

[0165] (Note 1) A method that is executed by at least one processor: A step of receiving a first video coding layer (VCL) network abstraction layer (NAL) unit of a first slice of an coded picture and a second VCL NAL unit of a second slice of the coded picture, wherein the first VCL NAL unit has a first VCL NAL unit type and the second VCL NAL unit has a second VCL NAL unit type different from the first VCL NAL unit type; A decoding step comprising the step of decoding the coded picture, the step of determining the picture type of the coded picture based on the first VCL NAL unit type of the first VCL NAL unit and the second VCL NAL unit type of the second VCL NAL unit, or based on an indicator received by the at least one processor indicating that the coded picture includes a mixed VCL NAL unit type; A method that includes this. (Note 2) The aforementioned decision-making step is: The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a trailing picture coded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit includes an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The method described in Appendix 1, which includes a step of determining based on the above. (Note 3) The aforementioned decision-making step is: The coded picture is a random-access decryptable reading (RADL) picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a RADL picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit includes an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The method described in Appendix 1, which includes a step of determining based on the above. (Note 4) The aforementioned decision-making step is: The coded picture is a Stepwise Temporal Sublayer Access (STSA) picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes an STSA picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain an immediate decoded refresh (IDR) picture-encoded slice, and The method described in Appendix 1, which includes a step of determining based on the above. (Note 5) The aforementioned decision-making step is: The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a stepwise temporal sublayer access (STSA) picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain a clean random access (CRA) picture-encoded slice. The method described in Appendix 1, which includes a step of determining based on the above. (Note 6) The aforementioned decision-making step is: The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a progressively decoded refresh (GDR) picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The method described in Appendix 1, which includes a step of determining based on the above. (Note 7) The aforementioned indicator is a flag; and The aforementioned decision-making step is: The coded picture is a trailing picture. The flag indicates that the coded picture includes mixed VCL NAL unit types. The method described in Appendix 1, which includes a step of determining based on the above. (Note 8) The aforementioned indicator is a flag; and The step of decoding the coded picture is: The temporal ID of the coded picture is 0. The flag indicates that the coded picture includes mixed VCL NAL unit types. The method described in Appendix 1, further comprising the step of determining based on the above. (Note 9) The aforementioned indicator is a flag; and The method according to Appendix 1, further comprising the step of receiving the flag in a picture header or slice header. (Note 10) The indicator is a flag, and the coded picture is in the first layer. The aforementioned method further The step of receiving the aforementioned flag; The additional coded picture in the second layer, which is a reference layer to the first layer, includes a mixed VCL NAL unit type. The step of determining, based on the flag indicating, that the coded picture includes mixed VCL NAL unit types; The method described in Appendix 1, including the method described in Appendix 1. (Note 11) It is a system: Memory configured to store computer programs; A system comprising at least one processor configured to receive at least one coded video stream, access the computer program code, and operate as directed by the computer code; The computer program code includes: A decoding code configured to cause at least one processor to decode a coded picture from the at least one coded video stream, Based on the first VCL NAL unit type of the first video coding layer (VCL) network abstraction layer (NAL) unit of the first slice of the coded picture, and the second VCL NAL unit type of the second VCL NAL unit of the second slice of the coded picture, or An indicator received by the at least one processor, based on an indicator indicating that the coded picture includes a mixed VCL NAL unit type, The decoding code includes a determination code configured to cause at least one processor to determine the picture type of the coded picture, The first VCL NAL unit type is a different system from the second VCL NAL unit type. (Note 12) The aforementioned decision code is, The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a trailing picture coded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit includes an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 13) The aforementioned decision code is, The coded picture is a random-access decryptable reading (RADL) picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a RADL picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit includes an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 14) The aforementioned decision code is, The coded picture is a Stepwise Temporal Sublayer Access (STSA) picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes an STSA picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain an immediate decoded refresh (IDR) picture-encoded slice. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 15) The aforementioned decision code is, The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a stepwise temporal sublayer access (STSA) picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain a clean random access (CRA) picture-encoded slice. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 16) The aforementioned decision code is, The coded picture is a trailing picture. The first VCL NAL unit type indicates that the first VCL NAL unit includes a progressively decoded refresh (GDR) picture-encoded slice, and The second VCL NAL unit type indicates that the second VCL NAL unit does not contain an immediate decoded refresh (IDR) picture-encoded slice or a clean random access (CRA) picture-encoded slice. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 17) The aforementioned indicator is a flag; and The aforementioned decision code is, The coded picture is a trailing picture. The flag indicates that the coded picture includes mixed VCL NAL unit types. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 18) The aforementioned indicator is a flag; and The aforementioned decision code is, The temporal ID of the coded picture is 0. The flag indicates that the coded picture includes mixed VCL NAL unit types. The system according to Appendix 11, configured to cause at least one processor to make a decision based on the above. (Note 19) The aforementioned indicator is a flag; and The system according to Appendix 11, wherein at least one processor is configured to receive the flag in a picture header or slice header. (Note 20) A non-temporary computer-readable medium for storing computer instructions, wherein, when executed by at least one processor, the computer instructions cause the at least one processor to decode an encoded picture from at least one encoded video stream. The aforementioned decoding process is Based on the first VCL NAL unit type of the first video coding layer (VCL) network abstraction layer (NAL) unit of the first slice of the coded picture, and the second VCL NAL unit type of the second VCL NAL unit of the second slice of the coded picture, or An indicator received by the at least one processor, based on an indicator indicating that the coded picture includes a mixed VCL NAL unit type, This includes determining the picture type of the coded picture, The first VCL NAL unit type is a different medium from the second VCL NAL unit type.

Claims

1. A method that is executed by at least one processor: Steps include receiving video data comprising a first video coding layer (VCL) network abstraction layer (NAL) unit of a first slice of coded picture, a second VCL NAL unit of a second slice of the coded picture, and a mixed flag; The steps include determining the type of the first VCL NAL unit in the first slice and the type of the second VCL NAL unit in the second slice based on the value of the mixed flag, and decoding the encoded picture; A method comprising a method in which mixing of VCL NAL units of intra-random access point (IRAP) pictures and VCL NAL units of stepwise temporal sublayer access (STSA) pictures is prohibited, and the value of the mixing flag being equal to a predetermined value means that the type of the first VCL NAL unit indicates the IRAP picture and the type of the second VCL NAL unit indicates a non-IRAP picture that is not the STSA picture.

2. A method according to claim 1, wherein the value of the mixed flag is equal to another predetermined value, indicating that the type of the first VCL NAL unit is the same as the type of the second VCL NAL unit.

3. The method according to claim 1, wherein if the value of the mixed flag is equal to a predetermined value, the type of the first VCL NAL unit is an immediate decoded refresh (IDR) picture or a clean random access (CRA) picture.

4. The method according to claim 1, wherein if the value of the mixed flag is equal to a predetermined value, the type of the second VCL NAL unit indicates a random-access decodeable reading (RADL) picture.

5. The method according to claim 1, wherein if the value of the mixed flag is equal to a predetermined value, the type of the second VCL NAL unit is a trailing picture or a progressively decoded refresh (GDR) picture.

6. The method according to claim 1, wherein the mixed flag is included in the picture header or slice header.

7. In the method according to claim 1, the type of the first VCL NAL unit of the first slice is indicated by a first "nal_unit_type" syntax element, and the type of the second VCL NAL unit of the second slice is indicated by a second "nal_unit_type" syntax element. The mixed flag is indicated by the syntax element "mixed_nalu_types_in_pic_flag" and the predetermined value is 1, in this method.

8. A computer program that causes at least one processor to perform the method according to any one of claims 1 to 7.

9. A method that is executed by at least one processor: The steps of generating a first video coding layer (VCL) network abstraction layer (NAL) unit for a first slice of a picture contained in video data, and a second VCL NAL unit for a second slice of the picture contained in the video data; Mixing of VCL NAL units of intra-random access point (IRAP) pictures and VCL NAL units of stepwise temporal sublayer access (STSA) pictures is prohibited, and the mixing flag is set to a predetermined value when the type of the first VCL NAL unit indicates the IRAP picture and the type of the second VCL NAL unit indicates a non-IRAP picture that is not the STSA picture, while the mixing flag is set to a different predetermined value when the type of the first VCL NAL unit is the same as the type of the second VCL NAL unit; The steps include signaling the mixed flag to a decoder that decodes the video data; Methods that include...

10. A method that is executed by at least one processor: The steps of generating a first video coding layer (VCL) network abstraction layer (NAL) unit for a first slice of a picture contained in video data, and a second VCL NAL unit for a second slice of the picture contained in the video data; Mixing of VCL NAL units of intra-random access point (IRAP) pictures and VCL NAL units of stepwise temporal sublayer access (STSA) pictures is prohibited, and the mixing flag is set to a predetermined value when the type of the first VCL NAL unit indicates the IRAP picture and the type of the second VCL NAL unit indicates a non-IRAP picture that is not the STSA picture, while the mixing flag is set to a different predetermined value when the type of the first VCL NAL unit is the same as the type of the second VCL NAL unit; The first VCL NAL unit, the second VCL NAL unit, and the video data including the mixed flag are transmitted to the decoder as a coded bitstream; Methods that include...

Images