Picture having mixed NAL unit type

Abstract

To disclose a video coding mechanism.SOLUTION: A mechanism contains a step of receiving a plurality of sub-pictures related to a picture and a bit stream containing a flag. Each sub-picture is contained in a plurality of video coding layer (VCL) network abstraction layer (NAL) unit. A value of the NAL unit type is similar to all of VCL NAL units related to the picture when the flag is set to a first value. When the flag is set to a second value, the value of a first NAL unit type related to the VCL NAL unit containing one or more of the sub-pictures of the picture is different from the value of a second NAL unit type related to the VCL NAL unit containing one or more of the sub-pictures of the picture. Each sub-picture is decoded on the basis of the value of the NAL unit type. Each sub-picture is transferred for displaying one part of a decoded video sequence.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application is a joint publication of Ye-Kui Wang et al., entitled "Support Of Mixed U.S. Provisional Patent Application No. 62, entitled "Embedded NAL Unit Types Within One Picture In Video Coding" / 816,749 and "Support Of Mixed U.S. Provisional Patent Application No. 62 / 8, entitled "NAL Unit Types Within One Picture In Video Coding" This application claims the benefit of US Provisional Patent Application No. 32,132, the entire disclosure of which is incorporated herein by reference. To be incorporated.

[0002] The present disclosure relates generally to video coding, and more particularly to a method for improving picture quality in video coding. This relates to coding subpictures of a picture. [Background technology]

[0003] The amount of video data required to render even a relatively short video can be significant. and that the data is streamed over a communications network having limited bandwidth capacity. even if it poses difficulties when the information should be Thus, video data is generally transmitted over modern communication networks. Because memory resources may be limited, the video is compressed before it is stored. The size of the video can also be an issue when it is stored on a storage device. Video compression devices are used to code video data before transmission or storage. The use of software and / or hardware at the source, thereby The compressed video image is then compressed to reduce the amount of data required to represent the video image. The data is received at the destination by a video decompression device which decodes the video data. Due to limited network resources and ever-increasing demand for higher video quality, This results in improved compression, increasing the compression ratio with little or no sacrifice in image quality. and thawing techniques are preferred. Summary of the Invention [Means for solving the problem]

[0004] In an embodiment, the present disclosure provides a method implemented in a decoder, comprising: A bitstream including a plurality of associated sub-pictures and flags is transmitted to a decoder receiver. Thus, the subpicture is encoded by a video coding layer (VCL: video coding layer). network abstraction layer (NAL) unit When the flag is set to a first value, the first NAL unit The process assumes that the type value is the same for all of the VCL NAL units associated with a picture. determining by the processor that the flag is set to a second value; The first NAL unit type for a VCL NAL unit that contains one or more of the subpictures The second NAL unit for a VCL NAL unit whose value contains one or more of the subpictures of a picture. determining by a processor a value of the first NAL unit type different from that of the second NAL unit type; one or more of the subpictures based on the value of the NAL unit type or the value of the second NAL unit type and decoding by a processor.

[0005] A picture may be partitioned into multiple sub-pictures. Such sub-pictures may be , which can then be coded into sub-bitstreams. The bitstreams may be merged into a bitstream for transmission to a decoder. For example, subpictures may be used for virtual reality (VR) applications. In this example, the user may only see a portion of the VR picture at any one time. More bandwidth can be allocated to sub-pictures that are more likely to be displayed, Subpictures that are unlikely to be shown may be compressed to improve coding efficiency. As possible, different sub-pictures may be transmitted at different resolutions. VideoStream is an intra-random access point (IRAP). An IRAP picture may be coded using an intra-predicted (int) picture. It is coded exponentially and can be decoded without reference to other pictures. Non-IRAP pictures may be coded by inter prediction, and other pictures Non-IRAP pictures can be decoded by referencing the IRAP picture. However, an IRAP picture can be decoded without reference to other pictures. Since the video sequence contains enough data to start decoding from the IRAP picture, IRAP pictures can be used within subpictures, This allows for dynamic resolution changes. Thus, the video system (e.g., for subpictures that are more likely to be seen (based on the user's current viewport) In order to transmit more IRAP pictures and further increase coding efficiency, Fewer IRAP pictures may be transmitted for less useful sub-pictures. The pictures are part of the same picture. Therefore, this method is similar to IRAP subpictures. Some video systems may result in pictures that contain both IRAP and non-IRAP subpictures. does not provide for handling mixed pictures that have both IRAP and non-IRAP regions. Whether a picture is mixed and therefore contains both IRAP and non-IRAP components Based on this flag, the decoder determines whether the picture / subpicture In order to properly decode and display the image, different sub-pictures must be treated differently when decoding. This flag may be stored in the PPS and is called mixed_nalu_types_in_pic_flag. Thus, the disclosed mechanism allows for the implementation of additional functionality. Additionally, the disclosed mechanism is useful when using subpicture bitstreams. It allows dynamic resolution change. Thus, the disclosed mechanism improves user experience. Supports lower resolutions when streaming VR videos without noticeable loss of quality. This allows multiple sub-picture bitstreams to be transmitted. The proposed mechanism increases the coding efficiency and therefore the encoder and decoder use of network resources, memory resources, and / or processing resources in Reduce.

[0006] Optionally, in any of the above aspects, another implementation of the aspect comprises: This specifies that the system contains a picture parameter set (PPS) that contains a flag.

[0007] Optionally, in any of the above aspects, another implementation of the aspect comprises: The value of the type indicates that the picture contains an Intra Random Access Point (IRAP) subpicture. The second NAL unit type value indicates that the picture contains a non-IRAP subpicture. It is stipulated that:

[0008] Optionally, in any of the above aspects, another implementation of the aspect comprises: The value of the type is a random access decodable leading picture. Instantaneous Decoding Refresh (IDR) with fresh (IDR_W_RADL), IDR with no leading picture (IDR_N_LP), or clean It is defined as equal to the clean random access (CRA) NAL unit type (CRA_NUT). Determine.

[0009] Optionally, in any of the above aspects, another implementation of the aspect further comprises: The value of the trailing picture NAL unit type (TRAIL_NAL_UNIT_TYPE) T), random access decodable leading picture NAL unit type (RADL_NUT), or or random access skip reading pictures (RASL) , and specifies that the NAL unit type is equal to the RASL_NUT (represented leading picture) NAL unit type.

[0010] Optionally, in any of the above aspects, another implementation of the aspect is to further include a step of adding a flag mixed_ It is specified that the format is nalu_types_in_pic_flag.

[0011] Optionally, in any of the above aspects, another implementation of the aspect refers to a PPS. If a picture has two or more of the VCL NAL units and the VCL NAL units have the same NAL unit type, When specifying that the two types of data do not have the same value of nal_unit_type, If lag is equal to 1, and the picture that references the PPS has one or more VCL NAL units, and the VCL When specifying that NAL units have the same value of nal_unit_type, mixed_nalu_types_ Specify in_pic_flag equal to 0.

[0012] In an embodiment, the present disclosure provides a method implemented in an encoder, comprising: determining by the processor whether the image contains multiple sub-pictures of different types; The bitstream encodes the subpictures of a picture and the subpictures of the picture into multiple VCL NAL units. and converting all VCL NAL units whose first NAL unit type value is associated with a picture into a VCL NAL unit. It is set to the first value when the pixels in the picture are the same in terms of units, and The first NAL unit type value for a VCL NAL unit that contains one or more of the following: a second NAL unit type value for a VCL NAL unit that contains one or more of the pictures; A flag set to a second value at a different time is encoded into the bitstream by the processor. and a processor coupled to the processor for transmitting the bitstream to a decoder. and storing by a memory provided with the

[0013] A picture may be partitioned into multiple sub-pictures. Such sub-pictures may be , which can then be coded into sub-bitstreams. The bitstreams may be merged into a bitstream for transmission to a decoder. For example, subpictures may be used for virtual reality (VR) applications. In this example, the user may only see a portion of the VR picture at any one time. More bandwidth can be allocated to sub-pictures that are more likely to be displayed, Subpictures that are unlikely to be shown may be compressed to improve coding efficiency. As possible, different sub-pictures may be transmitted at different resolutions. Video streams use Intra Random Access Point (IRAP) pictures. An IRAP picture may be coded by intra prediction. A non-IRAP picture can be decoded without reference to any other picture. It may be coded by vector prediction and restored by referring to other pictures. Non-IRAP pictures are condensed to a greater extent than IRAP pictures. contains enough data to be decoded without reference to other pictures. Therefore, the video sequence must start decoding from the IRAP picture. The RGB color space can be used within a subpicture, allowing dynamic resolution changes. Thus, the video system can determine the Transmit more IRAP pictures for subpictures that are more likely to be seen (by the user) ,To further improve coding efficiency, we use the ,subpictures that are unlikely to be seen. Fewer IRAP pictures may be transmitted, but the sub-pictures must be part of the same picture. Therefore, this method supports both IRAP and non-IRAP subpictures. Some video systems may result in a picture that contains IRAP and non-IRAP regions. There is no provision to handle mixed pictures that have both. This disclosure describes how pictures can be mixed, but It includes a flag indicating whether it contains both IRAP and non-IRAP components. Based on the flag, the decoder will decode and display the picture / subpicture appropriately. In addition, different sub-pictures may be treated differently when decoding. This may be stored in the mixed_nalu_types_in_pic_flag and may be called mixed_nalu_types_in_pic_flag. The mechanisms disclosed allow for the implementation of additional functionality. The system allows dynamic resolution changes when using subpicture bitstreams. Therefore, the disclosed mechanism does not significantly impair the user experience. Instead, it uses lower resolution subpicture bitstreams when streaming VR video. Thus, the disclosed mechanism allows for a code stream to be transmitted. Improves the coding efficiency and therefore reduces network resources in the encoder and decoder. , reducing the use of memory resources and / or processing resources.

[0014] Optionally, in any of the above aspects, another implementation of the aspect is to convert the PPS into a bit stream. and further comprising the step of encoding the flag into the PPS. Including etc.

[0015] Optionally, in any of the above aspects, another implementation of the aspect comprises: The value of the NAL unit type indicates that the picture contains an IRAP subpicture, and the second NAL unit type This specifies that a value of 0 indicates that the picture contains a non-IRAP sub-picture.

[0016] Optionally, in any of the above aspects, another implementation of the aspect comprises: This specifies that the value of this parameter type is equal to IDR_W_RADL, IDR_N_LP, or CRA_NUT.

[0017] Optionally, in any of the above aspects, another implementation of the aspect further comprises: This specifies that the value of this type is equal to TRAIL_NUT, RADL_NUT, or RASL_NUT.

[0018] Optionally, in any of the above aspects, another implementation of the aspect is to further include a step of adding a flag mixed_ It is specified that the format is nalu_types_in_pic_flag.

[0019] Optionally, in any of the above aspects, another implementation of the aspect refers to a PPS. A picture has two or more VCL NAL units, and the VCL NAL units have the nal_unit_type When specifying that mixed_nalu_types_in_pic_flag is equal to 1 and PPS has one or more VCL NAL units, and the VCL NAL unit is nal_u When specifying that they have the same value of nit_type, mixed_nalu_types_in_pic_flag is equal to 0. It is stipulated that

[0020] In an embodiment, the present disclosure provides a method for detecting a signal from a processor, a receiver coupled to the processor, and a method for detecting a signal from the processor. a memory coupled to the processor; and a transmitter coupled to the processor, The receiver, memory, and transmitter are configured to perform the method of any of the above aspects. This includes video coding devices.

[0021] In an embodiment, the present disclosure provides a code for use by a video coding device. A non-transitory computer-readable medium containing a computer program product, The program product, when executed by the processor, A method for performing any of the above-mentioned aspects, comprising: A non-transitory computer-readable medium containing computer-executable instructions is included.

[0022] In an embodiment, the present disclosure provides a method for generating a plurality of sub-pictures and flags associated with a picture. A receiving means for receiving a bitstream including a sub-picture, the sub-picture being A receiving means and a determining means, which are included in the NAL unit, and a flag is set to a first value. When the first NAL unit type value is set, all VCL NAL units related to the picture are and if the flag is set to a second value, The first NAL unit type value for a VCL NAL unit that contains one or more pictures is The second NAL unit for a VCL NAL unit that contains one or more of the subpictures of a picture. determining means for determining that the value of the first NAL unit type is different from the value of the first NAL unit type; to decode one or more of the subpictures based on a value of the second NAL unit type. and a decoding means for decoding the received signal.

[0023] Optionally, in any of the above aspects, another implementation of the aspect is a decoder comprising: It is provided that the method is further configured to perform the method of any of the above aspects.

[0024] In an embodiment, the present disclosure provides a method for determining whether a picture contains multiple sub-pictures of different types. a determining means for determining whether the sub-picture of the picture is a picture; The first NAL unit type value is It is set to the first value when it is the same for all VCL NAL units related to the picture. The first NAL unit is a VCL NAL unit that contains one or more of the subpictures of the picture. The unit type value relates to a VCL NAL unit that contains one or more of the subpictures of a picture. When the value of the second NAL unit type is different from the value of the second NAL unit type, the flag is set to the second value. encoding means for encoding the bit stream into a bit stream for transmission to a decoder; and storage means for storing the program.

[0025] Optionally, in any of the above aspects, another implementation of the aspect is further characterized in that the encoder: It is provided that the device is further configured to carry out the method of any of the above aspects.

[0026] For purposes of clarity, any one of the above-described embodiments may be considered as a new embodiment within the scope of the present disclosure. may be combined with any one or more of the other above-mentioned embodiments to produce stomach.

[0027] These and other features will become apparent from the following detailed description taken in conjunction with the accompanying drawings and claims. By understanding it will be more clearly understood.

[0028] In order to provide a more complete understanding of this disclosure, reference is now made to the accompanying drawings, in which like reference numerals represent like parts and in which: Reference is now made to the following brief description, which should be taken in conjunction with the detailed description. [Brief description of the drawings]

[0029] [Figure 1] 1 is a flow diagram of an exemplary method for coding a video signal. [Diagram 2] 1 is a schematic diagram of an example coding and decoding (codec) system for video coding. [Diagram 3] FIG. 1 is a schematic diagram illustrating an exemplary video encoder. [Figure 4] FIG. 2 is a schematic diagram illustrating an exemplary video decoder. [Diagram 5] FIG. 2 is a schematic diagram illustrating an example coded video sequence. [Figure 6] FIG. 1 is a schematic diagram showing multiple sub-picture video streams split from a virtual reality (VR) picture video stream. [Figure 7] 1 is a schematic diagram illustrating an example bitstream including pictures having mixed Network Abstraction Layer (NAL) unit types. [Figure 8] 1 is a schematic diagram of an example video coding device. [Figure 9] 1 is a flow diagram of an example method for encoding a video sequence including pictures having mixed NAL unit types in a bitstream. [Figure 10] 1 is a flow diagram of an example method for decoding a video sequence including pictures having mixed NAL unit types from a bitstream. [Figure 11] 1 is a schematic diagram of an example system for coding a video sequence including pictures having mixed NAL unit types in a bitstream. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Exemplary implementations of one or more embodiments are provided below, but are not limited to the disclosed systems and / or The method may be implemented using any number of techniques, whether currently known or in existence. It should be understood at the outset that the present disclosure may be implemented using any of the techniques described herein. Exemplary implementations shown below, including exemplary designs and implementations illustrated and described in FIG. The present invention should not be limited in any way to the aspects and techniques set forth herein, but should be construed as being in accordance with the full scope of equivalents of the appended claims. may be modified within the scope of those claims.

[0031] The following acronyms are used: Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Instantaneous Decode Refresh (IDR), Intra Random Access Point Int (IRAP), Least Significant Bit (LSB), Most Significant Bit (MSB), Network Abstraction Layer (NAL ), Picture Order Count (POC), Raw Byte Sequence Payload ( RBSP: Raw Byte Sequence Payload), Sequence Parameter Set (SPS), and Working Draft The draft (WD) is used herein.

[0032] Many video compression techniques reduce the size of video files with minimal loss of data. For example, video compression techniques can be used to reduce data redundancy in a video sequence. Spatial (e.g., intra-picture) prediction and temporal (e.g., inter-picture) prediction are used to reduce or eliminate The block-based video coding may include performing inter-picture (inter-picture) prediction. For example, a video slice (e.g., a video picture or a portion of a video picture) may be The video block may be partitioned into a tree block, a coding block, and a video block. coding tree block (CTB), coding tree unit (CTU), coding unit (CU) , and / or coding nodes. Video blocks in an I-slice that are mapped to a particular picture are stored in the I-slice. The inter-picture coding is done using spatial prediction relative to the reference sample. Video blocks in a unidirectionally predicted (P) or bidirectionally predicted (B) slice that are coded Spatial prediction or other reference samples relative to reference samples in neighboring blocks in the picture The samples are coded by using time prediction relative to the reference samples in the A picture may be referred to as a frame and / or an image, and a reference picture may be They may also be called reference frames and / or reference images. The residual data is the difference between the original block and the predicted block. Therefore, an inter-coded block represents the pixel difference between the predicted block and the The motion vectors point to blocks of reference samples forming the block to be coded. The predicted block is coded by residual data indicating the difference between the predicted block and the block. The blocks to be coded are coded by the intra-coding mode and the residual data. For further compression, the residual data is transformed from the pixel domain to the transform domain. These result in residual transform coefficients, which may be quantized. First, the quantized transform coefficients may be arranged in a two-dimensional array. The coefficients may be scanned to generate a one-dimensional vector of transform coefficients. Video coding may be applied to achieve even greater compression. The compression techniques are discussed in detail below.

[0033] To ensure that the encoded video can be accurately decoded, the video is The video signal is encoded and decoded according to a video coding standard. International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Moving Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10 Also known as Advanced Video Coding (AVC), ITU-T H.26 AVC includes High Efficiency Video Coding (HEVC), also known as MPEG-H Part 5 or MPEG-H Part 2. , Scalable Video Coding (SVC), Multi-view Video Coding Multiview Video Coding (MVC) and Multiview Video Coding Plus Depth (MVC+D) Multiview Video Coding plus Depth), and three-dimensional (3D) AVC (3D-AVC) and other extensions. HEVC includes Scalable HEVC (SHVC), Multi-View HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC Joint Video Experts Team (JVET) The erts team is developing a new video coding technology called Versatile Video Coding (VVC). The development of a decoding standard has begun. VVC is a description of the algorithm, a VVC Working Draft (WD), WD provides a description of the encoder side and reference software, including JVET-M1001-v6. Can be enjoyed.

[0034] A video coding system may use IRAP and non-IRAP pictures. The IRAP picture is a randomly generated picture for a video sequence. A picture coded by intra prediction that serves as an access point. In intra prediction, a block of a picture is predicted based on the difference between other blocks in the same picture. This is the same as for non-IRAP pictures that use inter prediction. In contrast, in inter prediction, a block in the current picture is predicted by a block in the current picture. The IR is coded by reference to other blocks in a different reference picture. AP pictures are coded without reference to any other pictures, so they are not the first Other pictures can be decoded without decoding. Therefore, the decoder can decode any IRAP picture. In contrast, decoding of a video sequence can start at a non-IRAP picture. A picture is coded with reference to other pictures, so that the decoder generally , decoding of a video sequence cannot start at a non-IRAP picture. AP picture refreshes the DPB because the IRAP picture is the start of the CVS and the CVS S does not refer to a picture in the previous CVS. The decoder further ensures that coding errors associated with inter prediction are propagated through the IRAP pictures. However, IRAP pictures can be used to prevent such errors. In terms of data size, IRAP pictures are significantly larger than non-IRAP pictures. The video sequence uses many non-IRAP It contains a 60 frame frame with fewer IRAP pictures interspersed among them. The CVS of a frame may contain one IRAP picture and 59 non-IRAP pictures.

[0035] In some cases, video coding systems are also referred to as 360-degree video. It may also be used to code virtual reality (VR) video, which allows a user to The viewport may contain a sphere of video content displayed as if the viewer were at the center of the sphere. Only a portion of the sphere, called the sphere, is visible to the user. A head-mounted display that selects and displays a sphere viewport based on the user's head movements. A head-mounted display (HMD), which is a device that physically exists in the virtual space depicted by the video, may also be used. To achieve this result, each picture in the video sequence is ,contains the entire sphere of video data for the corresponding instant.,However, a small part of the picture (even Only one part of the picture (for example, a single viewport) is displayed to the user. The rest of the picture is Different viewports are dynamically selected according to the user's head movements. Typically, the entire picture is transmitted so that it can be selected and displayed. This may lead to large video file sizes.

[0036] To improve coding efficiency, some systems split a picture into subpictures. A subpicture is a defined spatial region of a picture. Each subpicture is , which contains the corresponding viewport of the picture. Video may be encoded in more than one resolution. Each resolution is encoded into a different sub-bitstream. When streaming, the coding system uses the current video being used by the user. Merging sub-bitstreams into a bitstream for transmission based on a user port In particular, the current viewport can be obtained from the high resolution sub-bitstream. The unseen viewports are derived from the lower resolution bitstream. In this way, the highest quality video is displayed to the user and the lower quality video is If the user selects a new viewport, the lower resolution video is presented to the user. The decoder determines that the new viewport is a higher resolution video. The encoder may then request to receive the merged video accordingly. When an IRAP picture is reached, the decoder will update the higher resolution of the new viewport. This approach has a significant impact on the user's viewing experience. Greatly improve video compression without any negative impact.

[0037] One concern with the above approach is that the amount of time required to change the resolution is This is based on the length of time it takes to reach the RAP picture. The reader cannot start decoding a different video sequence at a non-IRAP picture. One technique to reduce such latency is to use more IRAP pins. However, this will increase the file size. To balance loading efficiency, different viewports / subpictures are For example, the IRAP picture may be included in a viewport that is more likely to be seen. A viewport may have more IRAP pictures than other viewports. In the context of soccer, a viewport that looks at the stands or ceiling is the Since it is less likely to be seen by such a viewport, the basket and / or center Viewports associated with the turquoise may contain IRAP pictures more frequently.

[0038] This approach leads to other problems. In particular, the subpicture containing the viewport is simply Different sub-pictures have IRAP pictures with different frequencies. In some cases, some of the pictures contain both IRAP and non-IRAP subpictures. Since pictures are stored in the bitstream by using NAL units, A NAL unit is a parameter set or slice of a picture and the corresponding A slice header is a storage unit that contains an entire picture. Thus, an access unit contains all of the NAL units associated with a picture. A NAL unit also contains a type that indicates the type of picture that contains the slice. In a video system, the number of frames per picture is limited to a single picture (e.g., contained in the same access unit). All related NAL units are expected to be of the same type. The unit's storage mechanism allows pictures to be stored as both IRAP and non-IRAP subpictures. When the function is used in conjunction with

[0039] Disclosed herein is a method for encoding both IRAP and non-IRAP subpictures. It is a mechanism for adjusting the storage of NALs to support pictures that contain This, in turn, allows for a VR scene with different IRAP subpicture frequencies for different viewports. In a first example, the present disclosure provides a method for enabling mixed picture video. For example, the flag indicates whether the picture is an IRAP subpicture. Based on this flag, the decoding process may indicate that the subpicture contains both IRAP and non-IRAP subpictures. The decoder must use different decoding methods to properly decode and display the picture / subpicture. This flag may be used in the Picture Parameter Set to handle subpictures of different types. This may be stored in the PPS and may be called mixed_nalu_types_in_pic_flag.

[0040] In a second example, disclosed herein is a method for indicating whether a picture is mixed. A flag. For example, the flag indicates whether a picture is an IRAP or non-IRAP subpicture. In addition, the flag may indicate that the mixed picture contains both the The picture is arranged to include exactly two NAL unit types, one IRAP type and one non-IRAP type. For example, a picture is a random access decodable leading picture. Instantaneous Decoding Refresh (IDR) with leading picture (IDR_W_RADL), IDR without leading picture (IDR_N_LP), or clean random access (CRA) NAL unit type (CRA_NUT) In addition, a picture may contain an IRAP NAL unit that contains only one of the following: Picture NAL unit type (TRAIL_NUT), a random-access decodable leading picture NAL unit type (RADL_NUT), or random access skip reading Non-IRAP NAL unit that contains only one of the picture (RASL) NAL unit types (RASL_NUT) Based on this flag, the decoder will select the appropriate picture / subpicture. Different sub-pictures may be treated differently when decoded in order to be decoded and displayed simultaneously. This flag may be stored in the PPS and is called mixed_nalu_types_in_pic_flag. Good too.

[0041] FIG. 1 is a flow diagram of an exemplary operational method 100 for coding a video signal. The video signal is encoded in an encoder. The encoding process uses various mechanisms. Compresses the video signal by reducing the video file size using The small file sizes allow for the same quality as compressed video while reducing the associated bandwidth overhead. The decoder then converts the compressed video file into Decodes the video file and reconstructs the original video signal for display to the end user. In general, the decoding process is carried out in a way that allows the decoder to consistently reconstruct the video signal. It faithfully mimics the encoding process to achieve

[0042] In step 101, a video signal is input to an encoder. The signal may be an uncompressed video file stored in memory. The video file is captured by a video capture device such as a video camera. The video file may be encoded to support live streaming of video. may contain both audio and video components. A component is a sequence of image frames that, when viewed in sequence, give the visual impression of motion. A frame contains light and a set of light components, referred to herein as luma components (or luma samples). , and a color called a chroma component (or color sample). In some instances, the frames may also include depth values ​​to support three-dimensional viewing. stomach.

[0043] In step 103, the video is partitioned into blocks. This involves subdividing pixels into square and / or rectangular blocks for compression. For example, High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2 In the image, the frame is first cut to a predefined size (e.g., 64 pixels by 64 The CTU is divided into coding tree blocks, which are blocks of lumens (pixels). The CTU is divided into blocks, which then undergo further coding. A code is then used to repeatedly subdivide the blocks until a configuration that supports the For example, the luma component of a frame may be represented as It may be subdivided to contain values ​​for relatively uniform lighting. For example, The chroma components may be subdivided until each block contains a relatively uniform color value. Therefore, the segmentation mechanism varies depending on the content of the video frame.

[0044] In step 105, the image blocks divided in step 103 are compressed. Various compression mechanisms are used to achieve this. For example, inter prediction and / or intra prediction. Inter prediction may be used. Inter prediction is a method for predicting the time course of an object in a typical scene in successive frames. The reference frame is therefore designed to take advantage of the fact that A block depicting an object in a frame does not need to be repeatedly shown in neighboring frames. In particular, objects such as tables may remain in a constant position over multiple frames. Thus, the table is shown once and adjacent frames are displayed by looking back at the reference frame. The pattern matching mechanism allows for multiple frames to be referenced. In addition, the moving object may be used to match the object, e.g. It may be represented across multiple frames due to object motion or camera motion. As a specific example, a video may show a car moving across the screen over several frames. A motion vector may be used to indicate such a movement. gives the offset from the object's coordinates in the frame to the object's coordinates in the reference frame. It is a two-dimensional vector. Therefore, inter prediction is performed for image blocks in the current frame. as a set of motion vectors indicating the offsets from the corresponding blocks in the reference frame It can be encoded.

[0045] Intra prediction encodes blocks within a common frame. Take advantage of the fact that minute and chrominance components tend to cluster together within a frame. For example, ,Some green areas of a tree tend to be located in the neighborhood of similar green areas. supports multiple directional prediction modes (e.g., 33 in HEVC), planar modes, and DC Use the directional (DC) mode. The directional mode uses the neighboring blocks in the corresponding direction for the current block. Planar mode indicates that the samples in the block are similar / identical to each other. Planar mode indicates that the blocks are a series of blocks along rows / columns. We show that a block (e.g., a plane) can be interpolated based on neighboring blocks at the ends of a row. In this case, the planar mode uses a relatively constant gradient of changing values ​​to differentiate between rows / columns. Shows smooth transitions of light / color. DC mode is used for boundary smoothing and blocks The mean value associated with the samples of all neighboring blocks associated with the angular direction of the directional prediction mode. Therefore, the intra prediction block is the same as the image block. Instead of the actual values, the blocks may be expressed as values ​​of various related prediction modes. The measurement block may represent the image block as a motion vector value instead of its actual value. In any case, the predicted block may not accurately represent the image block in some cases. All the differences are stored in the residual block. To further compress the file, A transformation may be applied to the blocks.

[0046] In step 107, various filtering techniques may be applied. In this case, the filter is applied by an in-loop filtering method. Block-based prediction results in blocky images being generated at the decoder. Furthermore, the block-based prediction method may encode a block and then The blocks that are filtered may be reconstructed for later use as reference blocks. The filtering method is noise suppression filter, deblocking filter, adaptive loop filter. , and apply a sample adaptive offset (SAO) filter iteratively to the block / frame These filters are designed to ensure that the encoded file can be reconstructed accurately. These filters also reduce the blocking artifacts. In subsequent blocks, the object is coded based on the reconstructed reference block. in the reconstructed reference block so that it is less likely to introduce artifacts Reduces artifacts.

[0047] Once the video signal has been segmented, compressed and filtered, the resulting digital The data is encoded into a bitstream in step 109. The data discussed above and the associated and any signaling data desired for the purpose of the present invention. For example, such data may be data, prediction data, residual blocks, and various The bitstream may contain flags such as The bitstream may also be broadcast to multiple decoders. The generation of the bitstream may be performed in an iterative process. Therefore, steps 101, 103, 105, 107, and 109 are The method may be performed sequentially and / or simultaneously across multiple programs and blocks. The order given is presented for clarity and ease of review and is intended to be consistent with the video coding process. The present invention is not intended to limit the process to any particular order.

[0048] The decoder receives the bitstream and begins the decoding process in step 111. In particular, the decoder uses an entropy decoding scheme to convert the bitstream into the corresponding The decoder converts the bit stream into a bit stream of syntax and video data in step 111. The syntax data from the frame stream is used to determine the partition for the frame. The division should match the result of the block division in step 103. Step 1 The entropy coding / decoding used in 11 is described below. The imager selects a block from several possible choices based on the spatial location of values ​​in the input image. Many choices are made during the compression process, including the choice of block partitioning method. Signaling the selection may use multiple bins. When calculating the bins, the bins are treated as binary values ​​that are variable (e.g., bit positions that may vary depending on the situation). Entropy coding is a technique that encoders can apparently do better in certain cases. It allows us to discard all options that do not fit and leave a set of acceptable options. Then, each allowable choice is assigned a codeword. The size is based on the number of allowable choices (e.g., one bin for two choices, one for four choices, Then the encoder generates a code for the selected option. This scheme encodes the codeword as a potential combination of all possible choices. A small subset of acceptable options as opposed to a unique choice from a larger set We define the size of the codeword as large as is desirable to uniquely represent these choices. The decoder then determines the set of allowable choices in the same way as the encoder. The selection is decoded by determining the set of allowable choices. The reader can read the codeword and determine the selection made by the encoder.

[0049] In step 113, the decoder performs the decoding of the block. In particular, the decoder The inverse transform is used to generate a residual block. The decoder then The corresponding prediction block is used to reconstruct the image block according to the classification. The block is a combination of the intra-predicted block and the inter-predicted block generated by the encoder in step 105. The reconstructed image block may then be processed in step 11. 1. Position within the frame of the reconstructed video signal according to the segmentation data determined in 1. The syntax for step 113 is also the same as the entropy code discussed above. The information may be signaled in the bitstream by the

[0050] In step 115, the reconstructed video signal is generated in a manner similar to that of step 107 of the encoder. Filtering is performed on the frames of the signal. For example, noise suppression filters, digital filters, The blocking filter, adaptive loop filter, and SAO filter are may be applied to the frame to remove artifacts. The video signal is then displayed at step 117 for viewing by the end user. It can be output to play.

[0051] FIG. 2 illustrates an exemplary coding and decoding (codec) system for video coding. 2 is a schematic diagram of a codec system 200 that supports the implementation of the operating method 100. The codec system 200 provides the functionality for encoding and decoding. The codec system 200 is generalized to depict components used in both. , receiving a video signal as discussed in relation to steps 101 and 103 of the method of operation 100. , resulting in a segmented video signal 201. The system 200 may be configured as described above in connection with steps 105, 107, and 109 of the method 100. When acting as a coder, the segmented video signal 201 is converted into a coded bit stream. When acting as a decoder, the codec system 200 performs the steps of the method of operation 100. From the bitstream, as discussed in connection with steps 111, 113, 115, and 117 The codec system 200 includes a general coder control component 211, which generates an output video signal. Transformation, scaling and quantization component 213, intra picture estimation component 215, A trap picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaler Transformation and inverse transformation components 229, filter control analysis components 227, in-loop filter configuration components 225, the decoded picture buffer component 223, and the header format and context Context adaptive binary arithmetic coding (CABAC) structure 2. Such components are combined as shown. The solid lines indicate the movement of data being encoded / decoded, while the dashed lines indicate other components. The components of the codec system 200 are The decoder may be a sub-synchronization of the components of the codec system 200. For example, the decoder may include an intra-picture prediction component 217, Motion compensation component 219, scaling and inverse transform component 229, in-loop filter component These components may include a decoded picture buffer component 223, a decoded picture buffer component 225, and a decoded picture buffer component 223. , as described below.

[0052] A segmented video signal 201 is segmented into blocks of pixels by a coding tree. The coding tree is a representation of the various segments of a captured video sequence. Use divide mode to subdivide blocks of pixels into smaller blocks of pixels These blocks can then be subdivided into smaller blocks. They may also be called nodes in the coding tree. A larger parent node may have smaller children. The number of times a node is sub-divided depends on the depth of the node / coding tree. The divided blocks are sometimes included in a coding unit (CU). For example, a CU may have a luma block, a red difference chroma (Cr) block, and the blue difference chroma (Cb) block in the corresponding syntax for CU. Depending on the partitioning mode used, To partition a node into two, three, or four child nodes, each with a shape that changes depending on the This may include binary trees (BT), ternary trees (TT), and quad trees (QT) used for partitioning. A video signal 201 is passed through a general coder control component 211 for compression, conversion, scaling and and quantization component 213, intra-picture estimation component 215, filter control analysis component 22 7, as well as to motion estimation component 221.

[0053] The general coder control component 211 converts the video sequence into a bitstream according to the constraints of the application. The method is configured to make decisions related to coding of the images of the sequence. The adaptive coder control component 211 optimizes the bit rate / bit stream size versus reconstruction quality. Such decisions are based on storage space / bandwidth availability and image resolution. The general coder control component 211 may also control the buffer address. To reduce underrun and overrun problems, the buffer size is adjusted based on the transmission rate. To manage these issues, the general coder control component 211 manages the segmentation, prediction, and filtering by other components of the The adaptive coder control component 211 adjusts the compression complexity to increase resolution and bandwidth usage. Dynamically increase or decrease the resolution and compression complexity to reduce bandwidth usage. Therefore, the general coder control component 211 may dynamically reduce the regeneration of the video signal. The codec system 200's features are designed to balance build quality and bit rate concerns. The general coder control component 211 controls the operation of the other components. The control data also generates parameters for decoding in the decoder. The header format is to be encoded into the bitstream to signal and CABAC component 231.

[0054] The segmented video signal 201 is fed to a motion estimation component 221 and a motion vector estimation component 222 for inter prediction. Also transmitted to compensation component 219. Frames or slices of segmented video signal 201 The motion estimation component 221 and the motion compensation component 222 may be divided into multiple video blocks. The composition element 219 selects one or more blocks in one or more reference frames to provide temporal prediction. CODEC The system 200 may, for example, determine an appropriate coding mode for each block of video data. Multiple coding passes may be performed to select the coding mode.

[0055] The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but generally For illustrative purposes, they are shown separately. The motion estimation performed by the motion estimation component 221 Motion vectors are the process of generating motion vectors, which estimate motion for video blocks. The displacement vector is, for example, the displacement of the object to be coded relative to the prediction block. A prediction block may represent a block that is coded in terms of pixel differences. A predicted block is also called a reference block. Such pixel differences may be calculated using the sum of absolute differences (SAD), sum of squared differences (SSD), or may be determined by other difference metrics. HEVC uses CTU, coding tree Uses several coded objects, including blocks (CTBs) and CUs. For example, a CTU can be split into CTBs, and then the CTBs are split into CUs. A CU can be divided into a CB for each CU. A CU is a prediction unit that contains prediction data for the CU. (PU) and / or a transform unit (TU) that contains the transformed residual data. The motion estimation component 221 uses the rate-distortion analysis as part of the rate-distortion optimization process. For example, the motion estimation configuration element generates motion vectors, PUs, and TUs. The element 221 includes a number of reference blocks for the current block / frame, a number of motion vectors, etc. The reference block, motion vector, etc., that has the best rate-distortion characteristics may be determined. The best rate-distortion performance is determined by the quality of the video reconstruction (e.g., compression Both the coding efficiency (e.g., the amount of data loss due to the coding) and the size of the final encoding Balance the two.

[0056] In some examples, the codec system 200 stores the decoded picture in the decoded picture buffer component 223. The values ​​may be calculated for sub-integer pixel locations of the stored reference picture. The video codec system 200 uses quarter pixel positions, eighth pixel positions, and the like of the reference picture. The values ​​of the pixel positions, or other fractional pixel positions, may be interpolated. Thus, the motion estimation component 221 estimates the full pixel and fractional pixel positions. It may perform position-related motion searches and output motion vectors with fractional pixel accuracy. The motion estimation component 221 compares the position of the PU with the position of the prediction block of the reference picture. The motion vector of the video block in the slice to be inter-coded with respect to the PU is The motion estimation component 221 calculates the motion vector for encoding. The motion is output as motion data to the header format and CABAC component 231, and the motion is motion compensated. Output to component 219.

[0057] The motion compensation performed by the motion compensation component 219 is determined by the motion estimation component 221. deriving or generating a prediction block based on the determined motion vector. Again, the motion estimation component 221 and the motion compensation component 219 may be The motion vector for the PU of the current video block may be received. Motion compensation component 219 may then find the predictive block to which the motion vector points. Then, the pixel values ​​of the prediction block are calculated from the pixel values ​​of the current video block being coded. A residual video block is formed by subtracting pixel values ​​to form pixel difference values. In general, the motion estimation component 221 performs motion estimation related to the luma component and The compensation component 219 is calculated based on the luma component for both the chroma and luma components. The prediction block and the residual block are transformed, scaled, and and forwarded to the quantization component 213.

[0058] The segmented video signal 201 is then subjected to an intra-picture estimation component 215 and an intra-picture estimation component 216. The motion estimation component 221 and the motion compensation component 219 are also transmitted to the motion prediction component 217. Similarly, the intra-picture estimation component 215 and the intra-picture prediction component 217 are , may be highly integrated, but are shown separately for conceptual purposes. The pixel estimation component 215 and the intra-picture prediction component 217 perform the inter-frame prediction as described above. Instead of the inter prediction performed by the motion estimation component 221 and the motion compensation component 219, Instead, the current block is intra-predicted with respect to blocks in the current frame. The intra picture estimation component 215 estimates the intra picture used to encode the current block. In some examples, the intra picture estimation component 215 determines the intra prediction mode. , among the multiple tested intra-prediction modes, select the appropriate one for encoding the current block. The selected intra-prediction mode is then used for encoding. The encoded data is forwarded to the header format and CABAC component 231 for processing.

[0059] For example, the intra picture estimation component 215 may be implemented using various tested intra prediction models. Rate-distortion analysis was used to calculate the rate-distortion values ​​for the modes tested. The rate-distortion analysis is performed by selecting the intra prediction mode having the best rate-distortion characteristics. In general, a coded block and the blocks that were coded to generate the coded block are The amount of distortion (or error) between the original uncoded block and the coded block Determine the bit rate (e.g., number of bits) used to generate the image. The picture estimation component 215 determines which intra prediction mode is the best for the block. The distortion and level for the various coded blocks are then calculated to determine which of the blocks exhibits the highest distortion value. Furthermore, the intra picture estimation component 215 calculates the rate distortion ratio from the Depth blocks of the depth map are extracted using the Depth Modeling Mode (DMM) based on the optimization (RDO). It may be configured to code.

[0060] The intra-picture prediction component 217, when implemented in an encoder, The prediction block is generated based on the selected intra-prediction mode determined by the pixel estimation component 215. When implemented in a decoder, the residual block is generated from the bitstream. The residual block is the prediction block represented as a matrix. The residual block contains the difference between the original and the original block. The residual block is then transformed, scaled and and quantization component 213. The picture prediction component 217 may operate on both the luma and chroma components.

[0061] The transform, scale and quantize component 213 further compresses the residual block. The transform, scaling and quantization component 213 performs discrete conversion on the residual block. Apply a transform such as the discrete sine transform (DCT), discrete sine transform (DST), or a similar transform, and Generates video blocks containing differential transform coefficient values. A transform, or other type of transform, may also be used. The transform converts the residual information into pixel values. Transformations, scaling and quantization Component 213 may, for example, be configured to scale the transformed residual information based on frequency. Such scaling is further configured as follows: different frequency information is quantified with different granularity. This involves applying a scale factor to the residual information so that the reconstructed Transformation, scaling and quantization configurations may affect the final visual quality of the video. The construction element 213 is further configured to quantize the transform coefficients to further reduce the bit rate. The quantization process reduces the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting the quantization parameter. In some examples, the transform, scale and quantize component 213 then performs quantization. The quantized transform coefficients may be quantized by a bit-wise interpolation. The header format and CABAC component 231 are then transferred to the CABAC component 231 for encoding into the data stream. do.

[0062] The scaling and inverse transform component 229 performs transformation and scaling to support motion estimation. Apply the inverse operation of the scaling and quantization component 213. Element 229 is a reference block that may be, for example, a prediction block for another current block. Inverse scanning is performed to reconstruct the residual block in the pixel domain for later use as a lock. Apply scaling, inverse transform, and / or inverse quantization. Or the motion compensation component 219 may use the motion estimation of a subsequent block / frame. Calculate the reference block by adding the residual block back to the prediction block corresponding to To reduce artifacts introduced during scaling, quantization, and transformation, To do this, a filter is applied to the reconstructed reference block. Artifacts result in inaccurate predictions when subsequent blocks are predicted (and and further artifacts).

[0063] The filter control analysis component 227 and the in-loop filter component 225 control the residual block and and / or apply filters to the reconstructed image blocks, e.g. scaling and the transformed residual block from the inverse transform component 229 reconstructs the original image block. from the intra-picture prediction component 217 and / or the motion compensation component 219 to The reconstructed image block may then be combined with the corresponding predicted block. In some cases, the filter may be applied to the residual block instead. As with the other components of FIG. 2, the filter control analysis component 227 and and in-loop filter component 225 may be highly integrated and implemented together, For conceptual purposes, they are shown separately. Filters applied to the reconstructed reference block is applied to a particular spatial region, and we examine how such a filter is applied. The filter control analysis component 227 includes a number of parameters for adjusting such a filter. Analyzing the reconstructed reference block to determine where the filter should be applied; Set the corresponding parameters. Such data is used as filter control data for encoding. The data is forwarded to the header format and CABAC component 231 as a filter in the loop. The composition element 225 applies such filters based on the filter control data. The deblocking filter, noise suppression filter, SAO filter, and adaptive loop filter are Such filters may include, depending on the example, (e.g., in the spatial / pixel domain (for a given pixel block) or in the frequency domain. may be used.

[0064] When acting as an encoder, the filtered reconstructed image blocks , the residual block, and / or the prediction block are used in motion estimation, as discussed above. The decoded picture is stored in the decoded picture buffer component 223 for later use. When operating in this manner, the decoded picture buffer component 223 is reconstructed and filtered. The decoded blocks are stored and transferred to the display as part of the output video signal. The picture buffer component 223 stores the predicted blocks, the residual blocks, and / or the reconstructed blocks. The memory device may be any memory device capable of storing the generated image blocks.

[0065] The header format and CABAC components 231 are used in various configurations of the codec system 200. Coded to receive data from an element and transmit such data to a decoder. In particular, the header format and the CABAC components 231 are for encoding control data such as general control data and filter control data. Generates various headers. In addition, prediction data, including intra prediction and motion data, The residual data in the form of quantized and reconstructed transform coefficients are all coded into the bitstream. The final bitstream is then processed to reconstruct the original segmented video signal 201. Contains all the information desired by a decoder. Such information includes intra-prediction modes index table (also called codeword mapping table), Definition of coding context for the frame, indication of most likely intra prediction mode Such data may also include the encoding, the addressing, and an indication of partition information. For example, the information may be encoded using trope coding. Context adaptive variable length coding (CAVLC) g), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC) based context-adaptive binary arithmetic coding), Probability Interval Partition Entropy (PIPE: probability interval partitioning entropy) coding, or another form of entropy The image may be encoded using a coding technique. The encoded bitstream is then sent to another device (e.g. a video decoder). The information may be stored in a hard disk or archived for later transmission or retrieval.

[0066] FIG. 3 is a block diagram illustrating an example video encoder 300. may perform the encoding functions of the codec system 200 and / or the steps of the operating method 100. may be used to perform steps 101, 103, 105, 107, and / or 109. The encoder 300 segments the input video signal and outputs a signal substantially similar to the segmented video signal 201. The segmented video signal 301 is then The encoded data is compressed and encoded into a bitstream by components of the encoder 300.

[0067] In particular, the segmented video signal 301 is subjected to an intra-picture prediction scheme for intra prediction. The intra-picture prediction component 317 performs intra-picture estimation. The image prediction component 215 and the intra-picture prediction component 217 may be substantially similar to the image prediction component 215 and the intra-picture prediction component 217. The decoded video signal 301 is based on reference blocks in the decoded picture buffer component 323. For inter prediction, the motion compensation component 321 also The motion estimation component 221 and the motion compensation component 219 may be substantially similar to the intra-frame motion estimation component 221 and the motion compensation component 219. The prediction and residual blocks from the picture prediction component 317 and the motion compensation component 321 are The block is forwarded to the transform and quantization component 313 for transforming and quantizing the residual block. The transform and quantization component 313 is a transform, scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction The measurement block is then processed using an entropy coding structure for coding into a bitstream. The entropy coding configuration element 331 receives the entropy coding data and forwards it (together with associated control data) to the entropy coding configuration element 331. Element 331 may be substantially similar to the header format and CABAC component 231. .

[0068] Also, the transformed and quantized residual block and / or the corresponding prediction block are Transformation and quantification to reconstruct a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 then transfers the inverse transform and quantization component 329 to the inverse transform and quantization component 313. Component 329 may be substantially similar to the scaling and inverse transform component 229 . The in-loop filter of the in-loop filter component 325 may be implemented as a residual block and / or a filter for the in-loop filter component 325 depending on the example. or to the reconstructed reference block. , substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may be associated with the in-loop filter component 225. As discussed above, multiple filters may be included. The filtered block is then The block is then stored in a decoded picture buffer for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 stores the decoded picture buffer. The buffer component 223 may be substantially similar to the buffer component 223 .

[0069] 4 is a block diagram illustrating an example video decoder 400. Implementing the decoding function of the codec system 200 and / or step 1 of the operating method 100 11, 113, 115, and / or 117. The decoder 400 may be For example, a bitstream may be received from the encoder 300 and displayed to an end user. A reconstructed output video signal is generated based on the bitstream to

[0070] The bitstream is received by the entropy decoding component 433. The decoding component 433 may be CAVLC, CABAC, SBAC, PIPE coding, or other encoding. The decoder may be configured to implement an entropy decoding scheme, such as an entropy coding technique. For example, the entropy decoding component 433 may encode the bitstream as a codeword. Use the header information to provide context for interpreting further data returned. The decoded information may include general control data, filter control data, partition information, motion information, and the like. The video data, such as the input data, the prediction data, and the quantized transform coefficients from the residual block, The quantized transform coefficients are then used to decode the residual block. Transformation and Quantization The component 429 may be similar to the inverse transform and quantize component 329 .

[0071] The reconstructed residual block and / or predicted block are based on an intra prediction operation. The image blocks are then forwarded to the intra-picture prediction component 417 for reconstruction into image blocks. The intra-picture prediction component 417 is a combination of the intra-picture estimation component 215 and the intra-picture In particular, the intra-picture prediction structure may be substantially similar to the intra-picture prediction structure 217. The prediction mode is used to identify a reference block in the frame, and an intra prediction element 417 is used to perform intra prediction. We apply the residual block to the result to reconstruct the reconstructed image block. The intra-predicted image block and / or residual block and the corresponding inter-predicted image block and / or residual block are The measured data is fed to the decoded picture buffer component 423 via the in-loop filter component 425. These are transferred to the decoded picture buffer component 223 and the in-loop filter component 224, respectively. The in-loop filter component 425 may be substantially similar to the in-loop filter component 225. Filter the reconstructed image block, the residual block, and / or the prediction block. Such information is stored in the decoded picture buffer component 423. The reconstructed image block from the buffer component 423 is motion compensated for inter prediction. The motion compensation component 421 is forwarded to the motion estimation component 221 and / or The motion compensation component 421 may be substantially similar to the motion compensation component 219. In particular, the motion compensation component 421 A motion vector from the reference block is used to generate a prediction block and reconstruct the image block. We apply the residual block to the result to obtain the , also forwarded to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 may store additional reconstructed image blocks. The reconstructed image blocks are then reconstructed into frames according to the partition information. Such frames may also be arranged in a sequence. The resulting signal is output to a display as a reconstructed output video signal.

[0072] FIG. 5 is a schematic diagram illustrating an example CVS 500. For example, the CVS 500 may include a The digital signal is encoded by the digital signal system 200 and / or an encoder such as the encoder 300. In addition, the CVS 500 may be implemented as a codec system 200 and / or a decoder 400, etc. The CVS 500 may be used to decode the coded pictures in decoding order 508. The decoding order 508 is the order in which the pictures are placed in the bitstream. The pictures in the CVS 500 are then output in a presentation order 510. The presentation order 510 represents the resulting This is the order in which pictures should be displayed by a decoder to properly display the video being For example, pictures in a CVS 500 may be generally arranged in presentation order 510. However, certain pictures may be overridden by similar pictures to support inter-prediction, for example. moved to different locations to improve coding efficiency by placing them closer together Shifting such pictures in this manner results in a decoding order 508. In the example shown, the pictures are indexed in decoding order 508 from 0 to 4. In the presentation order 510, the pictures with indexes 2 and 3 are displayed before the picture with index 0. It has been moved in front of Kucha.

[0073] The CVS 500 includes an IRAP picture 502. The IRAP picture 502 is a random access map for the CVS 500. A picture coded by intra prediction serves as an access point. In particular, blocks of the IRAP picture 502 are referenced by other blocks of the IRAP picture 502. An IRAP picture 502 is coded without reference to any other picture. Since the picture is filtered, it can be decoded without first decoding any other pictures. , the decoder can start decoding the CVS 500 at the IRAP picture 502. For example, the IRAP picture 502 may cause the DPB to be refreshed. The pictures presented later are IRAP pictures 502 for inter prediction (e.g., pictures There is no need to rely on the previous picture (index 0). Therefore, the picture buffer is The IRAP picture 502 may be refreshed when it is decoded. This is related to inter prediction. Since the coding error cannot be propagated through the IRAP picture 502, all This has the effect of preventing such errors. The IRAP picture 502 is a picture of various types. For example, an IRAP picture may be coded as IDR or CRA. IDR is an intraconverter that starts a new CVS 500 and refreshes the picture buffer. The CRA is responsible for starting a new CVS 500 or Intra-coder that acts as a random access point without refreshing the buffer In this way, the leading picture 5 related to the CRA is 04 may refer to a picture preceding the CRA, while the leading picture 50 associated with the IDR 4 does not need to refer to the picture before the IDR.

[0074] CVS 500 also includes various non-IRAP pictures. These are leading pictures 504 and The leading pictures 504 are IRAP pictures in decoding order 508. A picture that is positioned after the IRAP picture 502 in the presentation order 510 but before the IRAP picture 502 in the presentation order 510. The trailing picture 506 is an IRAP picture in both decoding order 508 and presentation order 510. The leading picture 504 and the trailing picture 505 are positioned after the leading picture 502. 6 are both coded with inter prediction. The code is generated by referring to the IRAP picture 502 or a picture positioned after the IRAP picture 502. Therefore, the trailing picture 506 is always decoded after the IRAP picture 502 is decoded. The leading picture 504 can be decoded by random access skip leading. may include Random Access Decodable Reading (RASL) and Random Access Decodable Reading (RADL) pictures. The RASL picture is coded with reference to the picture before the IRAP picture 502. However, it is coded at a position after the IRAP picture 502. Since the decoding is based on the IRAP picture 502, the decoder starts decoding at the IRAP picture 502. Therefore, the RASL picture cannot be decoded even if the IRAP picture 502 is a random access point. When used as a digital input, it is skipped and not decoded. However, RASL pictures are The coder randomly accesses the previous IRAP picture (before index 0, not shown). When used as a point, it is decoded and displayed. RADL pictures are IRAP pictures 50 2 and / or the IRAP picture 502, but is coded with reference to the picture following the IRAP picture 502 The RADL picture is positioned before the IRAP picture 502 in the order 510. Since the IRAP picture 502 is a random access point, and can be displayed.

[0075] Pictures from the CVS 500 may be stored in respective access units. A picture may be partitioned into slices, which are contained in NAL units. A NAL unit may represent a parameter set or a slice of a picture and the corresponding A NAL unit is a storage unit that includes a rice header. For example, from IRAP picture 502, The slice is an IDR (IDR_W_RADL) NAL unit with RADL, a leading picture, and It may be included in an IDR (IDR_N_LP) NAL unit, a CRA NAL unit, etc. The L unit is an IDR picture in which an IRAP picture 502 is associated with a RADL leading picture 504. The IDR_N_LP NAL unit indicates that the IRAP picture 502 is a leading CRA NAL unit 504. is a CRA picture that the IRAP picture 502 may be associated with a leading picture 504. Slices of non-IRAP pictures may also be placed in NAL units. For example, a slice of the trailing picture 506 is A trailing picture NAL unit tag indicating that the picture is a predictively coded picture. The slices of the leading picture 504 may be arranged in the corresponding The picture corresponds to a type of inter-predictive coded leading picture 504. RASL NAL unit type (RASL_NUT) and / or RADL NAL unit type (RADL_NUT) A NAL unit may contain a slice of a picture within the corresponding NAL unit. By nulling, the decoder determines the appropriate decoding medium to apply to each picture / slice. The mechanism can be easily determined.

[0076] FIG. 6 shows a plurality of sub-picture video streams split from a VR picture video stream 600. FIG. 6 is a schematic diagram showing streams 601, 602, and 603. Each of the video streams 601 to 603 and / or the VR picture video stream 600 is coded in the CVS 500. Therefore, the sub-picture video streams 601 to 603 and / or The VR picture video stream 600 may be generated by the codec system 200 according to the method 100 and / or may be encoded by an encoder such as the encoder 300. The video streams 601 to 603 and / or the VR picture video stream 600 are It may be decoded by system 200 and / or a decoder such as decoder 400.

[0077] The VR picture video stream 600 includes a number of pictures presented over time. VR is a method for coding a sphere of video content that allows the user to appear as if they were in the center of the sphere. Each picture contains the entire sphere. On the other hand, the viewport and Only a portion of the picture known as the A head-mounted display that selects and displays a sphere viewport based on the user's head movements. A head-mounted display (HMD) may be used, which is a device that allows the user to physically enter the virtual space depicted by the video. To achieve this result, each picture in the video sequence contains the entire sphere of video data for the corresponding instant in time. However, it may only contain a small part of the picture (e.g. Only one part of the picture (e.g., a single viewport) is displayed to the user. The rest of the picture is The different viewports are dynamically changed according to the user's head movements. Generally, the entire picture is transmitted so that it can be selected and displayed.

[0078] In the illustrated example, the pictures of the VR picture video stream 600 are Each picture can be subdivided into sub-pictures based on the port. and the corresponding subpictures are assigned a temporal position (e.g., picture The sub-picture video streams 601 to 603 are arranged so that the subdivision is consistent over time. Such a consistent subdivision is generated when the subpicture video is applied to the Streams 601 to 603 are generated, and each stream has a predetermined size, shape, and VR picture. A set of sub-pictures of spatial location relative to a corresponding picture in the video stream 600. Furthermore, a set of sub-pictures in the sub-picture video streams 601-603 may be Therefore, the sub-picture video streams 601 to 603 are located at different positions in time. The pictures may be aligned in the time domain based on their temporal positions. Sub-pictures from the intermediate sub-picture video streams 601-603 are Based on predefined spatial positions, the VR picture video stream 600 is reconstructed. In particular, the sub-picture video streams 601-603 can be merged in the spatial domain. Each of these sub-bitstreams may be encoded into a separate sub-bitstream. When merged together, the frames result in a bitstream that contains the entire set of pictures over time. The resulting bitstream is then rendered according to the user's currently selected viewport. The image may be decoded based on the received image and transmitted to a decoder for display.

[0079] One of the problems with VR video is that all of the subpicture video streams 601 to 603 are high-resolution. The quality (e.g., high resolution) of the image may be transmitted to the user. This is because the decoder Dynamically selects the user's current viewport and outputs the corresponding sub-picture video stream 60 It allows the display of subpictures 1 to 603 in real time. However, the user For example, only one viewport sees the subpicture video stream 601. On the other hand, the sub-picture video streams 602-603 are discarded. Therefore, transmitting the sub-picture video streams 602-603 at high quality requires a large amount of bandwidth. To improve coding efficiency, VR video is split into multiple video streams. Each video stream 600 may be encoded into a different quality / resolution. In this way, the decoder receives the current sub-picture video stream 601 In response, the encoder (or intermediate slicer (intermediate slicer) Ediate slicer (or other content server) can stream higher quality video6 00 and a higher quality sub-picture video stream 601 from A lower quality sub-picture video stream 602-603 from stream 600 can be selected. The encoder then generates such sub-bitstreams for transmission to the decoder. In this way, the decoded bitstream can be merged into the complete encoded bitstream. The reader receives a series of pictures and determines whether the current viewport is of higher quality and the other The viewport is of lower quality. Furthermore, the highest quality subpicture is When the image is not being displayed, the lower quality subpicture is generally displayed to the user. are discarded, this is a trade-off between functionality and coding efficiency.

[0080] When the user shifts his / her attention from the sub-picture video stream 601 to the sub-picture video stream 602, When shifting the current sub-picture video stream 602, the decoder determines that the new current sub-picture video stream 602 is of higher quality. The encoder then adjusts the merging mechanism accordingly. As mentioned above, the decoder can change the new CVS 500 only at the IRAP picture 502. Therefore, the sub-picture video stream 602 may start decoding the IRAP picture / sub-picture video stream 602. The IRAP picture is then displayed at a lower quality until the subpicture is reached. A higher quality version of the video stream 602 is decoded. This technique allows the video compression to be adjusted without adversely affecting the user's viewing experience. This significantly improves shrinkage.

[0081] One concern with the above approach is the amount of time required to change the resolution. It is based on the length of time it takes to reach an IRAP picture in the video stream. indicates that the decoder should be able to read different versions of the subpicture video stream in a non-IRAP picture. This is because the decoding of the program 602 cannot be started. One approach to this is to include more IRAP pictures. To balance functionality and coding efficiency, different The viewport / subpicture video streams 601-603 are IRAP pictures at different frequencies. For example, viewport / subpicture video that is more likely to be seen The video streams 601 to 603 are the other viewport / subpicture video streams 601 to 60 For example, in the context of basketball, The viewport / subpicture video streams 601 to 603 are used to view the stand or ceiling. Such viewport / subpicture videos are less likely to be seen by Viewpoints related to basketball and / or center court rather than streams 601-603 Even if the intermediate / subpicture video streams 601 to 603 contain IRAP pictures more frequently, good.

[0082] This approach leads to further problems, especially when using sub-picture video streams that share POC. The sub-pictures from streams 601-603 are part of a single picture. Slices from a picture are included in NAL units based on the picture type. In a video coding system, all NAL units related to a single picture are , are constrained to contain the same NAL unit types. When the frames 601 to 603 have IRAP pictures with different frequencies, some of the pictures are IRAP sub-pictures. This includes both IRAP and non-IRAP subpictures, since each single picture is This violates the constraint that only NAL units of the same type should be used.

[0083] This disclosure relates to a method for determining whether all NAL units for a slice in a picture have the same NAL unit type. This problem is addressed by removing the constraint that you must use a picture. By removing this restriction, the access unit An event may contain both IRAP and non-IRAP NAL unit types. Furthermore, pictures / access units are classified as IRAP and non-IRAP NAL unit types. A flag may be coded to indicate when the mix includes a mix with a smear. The mixed NAL unit types in picture flag )(mixed_nalu_types_in_pic_flag). In addition, a single mixed picture / access unit A packet contains only one type of IRAP NAL unit and one type of non-IRAP NAL unit. This means that unintended NAL units may be Prevent type mixing from occurring. If such mixing were allowed, the decoder The code must be designed to manage such a mix. Eliminates the hardware complexity required without adding any additional benefit to the process For example, a mixed picture can be selected from IDR_W_RADL, IDR_N_LP, or CRA_NUT. In addition, a mixed picture may contain one type of IRAP NAL unit, TRAIL Contains one type of non-IRAP NAL unit selected from RASL_NUT, RADL_NUT, and RASL_NUT. An exemplary implementation of this scheme is discussed in more detail below.

[0084] FIG. 7 illustrates an example bitstream containing pictures with mixed NAL unit types. 7 is a schematic diagram of a bitstream 700. For example, the bitstream 700 may be a codec according to the method 100. 200 and / or decoder 400 for decoding by codec system 200 and / or decoder 400. The bitstream 700 may be generated by the encoder 300 and / or the encoder 300. A VR picture merged from multiple sub-picture video streams 601 to 603 having different video resolutions. Each sub-picture video stream may include a sub-picture video stream 600, each of which may include a different space. Includes CVS 500 in intermediate locations.

[0085] The bitstream 700 includes a sequence parameter set (SPS) 710, a number of picture parameters, The SPS 71 includes a meter set (PPS) 711, a number of slice headers 715, and image data 720. 0 is a sequence common to all pictures in the video sequence contained in the bitstream 700. Such data includes picture sizing, bit depth, coding This may include parameters for the picture rendering tool, bit rate constraints, etc. Each picture in a video sequence therefore contains parameters that apply to the entire image. , PPS 711. Each picture may refer to a PPS 711, but in some cases, Note that a single PPS 711 may contain data for multiple pictures. For example, multiple similar pictures may be coded with similar parameters. In such cases, a single PPS 711 may contain data for such similar pictures. PPS 711 specifies the coding tools available for slices within a corresponding picture. The slice header 715 may indicate the size of the picture, the frame rate, the quantization parameters, offsets, etc. It contains parameters specific to each slice in a video sequence. There may be one slice header 715 for each slice. The slice header 715 includes slice type information , Picture Order Count (POC), Reference Picture List, Prediction Weights, Tile Entry Points It may also include tile entry points, deblocking parameters, etc. Note that da 715 may also be called a tile group header depending on the context. .

[0086] The image data 720 may be video data encoded by inter prediction and / or intra prediction. The data includes the corresponding transformed and quantized residual data. The sequence includes a number of pictures 721 coded as image data 720. 721 is a single frame of a video sequence and therefore generally The subpicture 723 is displayed as a single unit when the sequence is displayed. The picture 721 may be displayed to implement certain technologies such as virtual reality. , PPS 711. A picture 721 may contain sub-pictures 723, tiles, and / or A sub-picture 723 may be divided into slices. A subpicture is a spatial region of a picture 721 that is consistently applied to all subpictures. The image 723 may be displayed by an HMD in a VR context. A sub-picture 723 is obtained from the sub-picture video streams 601 to 603 of the corresponding resolution. The Subpicture 723 may reference the SPS 710. The slice 725 is referred to as the tile group that contains the tiles. The tile group of a file references the slice header 715. A slice 725 is a single NAL unit. An integer number of complete tiles or an integer number of tiles in a picture 721 that are exclusively included in the unit Thus, a slice 725 may be defined as a row of consecutive complete CTUs. The CTU / CTB is further divided into a coding unit and / or a coding tree. The coding blocks are then further divided into coding blocks. It can be encoded / decoded by:

[0087] The parameter sets and / or slices 725 are coded into NAL units. An AL unit is a syntactic structure that contains an indication of the type of data that follows, and and inserting emulation prevention bytes here and there as necessary. It may be defined as a byte containing the data in the form of the entered RBSP. A NAL unit is a parameter set or slice 725 of a picture 721 and the corresponding slice In particular, the VCL NAL unit 740 is a storage unit that contains a slice header 715 of a picture 721. 7. A non-VCL NAL unit includes a slice header 715 and a corresponding slice header 725. Unit 730 includes parameter sets such as SPS 710 and PPS 711. For example, SPS 710 and PPS 711 are both SPS NAL unit type (SPS_NUT) 731, which is a non-VCL NAL unit 730, and PPS NAL units They may each be included in type (PPS_NUT) 732.

[0088] As described above, an IRAP picture, such as IRAP picture 502, is included in an IRAP NAL unit 745. Non-IRAP pictures such as leading picture 504 and trailing picture 506 are , non-IRAP NAL units 749. In particular, IRAP NAL units 745 are not included in IRAP pictures or A non-IRAP NAL unit is any NAL unit that contains a slice 725 obtained from a picture or a subpicture. The AL unit 749 may process any picture that is not an IRAP picture or a subpicture (e.g., Any slices 725 taken from the leading and trailing pictures IRAP NAL unit 745 and non-IRAP NAL unit 749 are both Since both contain slice data, they are both VCL NAL units 740. In this case, IRAP NAL unit 745 is an IDR or RADL picture without a leading picture. The slice 725 from the IDR associated with the channel is treated as an IDR_N_LP NAL unit 741 or an IDR_w_RADL NAL unit. In addition, the IRAP NAL unit 745 may be included in the CRA picture. These slices 725 may be included in the CRA_NUT 743. Unit 749 reads the slash from a RASL picture, a RADL picture, or a trailing picture. The device 725 may be included in the RASL_NUT 746, the RADL_NUT 747, or the TRAIL_NUT 748, respectively. In an exemplary embodiment, the complete list of possible NAL units is stored in the NAL unit type The results are sorted by and shown below. [Table 1A] [Table 1B] [Table 1C]

[0089] As mentioned above, a VR video stream is a sub-picture stream with IRAP pictures at different frequencies. This may include a filter 723, which is more sensitive to spatial regions that are less likely to be seen by the user. Fewer IRAP pictures are used, for spatial regions that users are likely to look at frequently. This allows more IRAP pictures to be used. Spatial regions that are likely to return frequently can be quickly adjusted to a higher resolution. The method results in a picture 721 that contains both IRAP NAL units 745 and non-IRAP NAL units 749. When this happens, picture 721 is called a mixed picture. This state is called intra-picture mixed NAL unit. The number of bits per 16 bits may be signaled by the mixed_nalu_types_in_pic_flag 727. The mixed_nalu_types_in_pic_flag 727 may be set in the PPS 711. _types_in_pic_flag 727 indicates that each picture 721 that references a PPS 711 consists of two or more VCL NAL units. 740, and the VCL NAL units 740 have the same value of NAL unit type (nal_unit_type). To specify that no mixed_nalu_type is to be used, it may be set equal to 1. s_in_pic_flag 727 indicates that each picture 721 that references a PPS 711 includes one or more VCL NAL units 740 and all VCL NAL units 740 of each picture 721 that references the PPS 711 are of type nal_unit_type When both have the same value of

[0090] In addition, when mixed_nalu_types_in_pic_flag 727 is set, the subpictures of picture 721 One or more VCL NAL units 740 in the chunk 723 all have a first specification of the NAL unit type. and all other VCL NAL units 740 in picture 721 have a value of NAL unit type Constraints may be used that have different second specific values. For example, the constraint may be a mixture A picture 721 contains a single type of IRAP NAL unit 745 and a single type of non-IRAP NAL unit. For example, picture 721 may need to include one or more I DR_N_LP NAL unit 741, one or more IDR_w_RADL NAL units 742, or none Any combination of such IRAP NAL units 745 may contain one or more CRA_NUTs 743. In addition, a picture 721 may contain one or more RASL_NUTs 746, one or more RADL_NUT 747 or one or more TRAIL_NUT 748, A packet cannot contain any combination of such IRAP NAL units 745.

[0091] In the exemplary implementation, the picture type is used to define the decoding process. Such a process may include, for example, identifying pictures by picture order count (POC). Deriving information,marking the status of reference pictures in the Decoded Picture Buffer (DPB),D The picture includes all the coded pictures. or its subparts may be identified by a type based on the NAL unit type that contains it. In some video coding systems, the picture type is instantaneous decoding refresh (I Other video coding systems may include IDR (Intermediate Distributed Random Access) pictures and non-IDR pictures. In the , picture type is trailing picture, temporal sublayer access (TSA: te mporal sub-layer picture, step-wise temporal sub-layer access (STSA) sub-layer access) pictures, Random Access Decodable Reading (RADL) pictures, Random Access Skip Reading (RASL) Pictures, Broken Link Access (BLA : broken-link access) pictures, instant random access pictures, and clean run Such picture types may include dumb-access pictures. whether it is a sub-layer referenced picture or a sub-layer non-referenced picture It may be further distinguished based on whether it is a sub-layer non-referenced picture. A BLA picture is a BLA with a leading picture, a BLA with a RADL picture, and and BLA, which has no leading pictures. IDR pictures are , and further distinguished as IDRs with RADL pictures and IDRs without leading pictures. This may be done.

[0092] Such picture types are used to implement various video-related functions. For example, an IDR, BLA, and / or CRA picture may implement an IRAP picture. An IRAP picture may provide the following features / benefits: The presence of a picture may indicate that the decoding process can start from that picture. This function ensures that the decoding process can proceed as long as an IRAP picture is present at that position in the bitstream. This allows the implementation of a random access feature that starts at a specified position. The presence of an IRAP picture does not necessarily coincide with the beginning of a bitstream. A coded picture that starts with an IRAP picture, excluding the IRAP picture, is placed before the IRAP picture. Redirect the decoding process so that the image is coded without reference to any placed pictures. Therefore, IRAP pictures located in the bitstream are refreshed. Therefore, the coding sequence positioned before the IRAP picture stops the propagation of coding errors. Decoding errors in the pictures that follow the IRAP picture in decoding order are reported through the IRAP picture. cannot be propagated directly to the picture.

[0093] IRAP pictures provide various features but come at a cost to compression efficiency. Therefore, the presence of IRAP pictures can cause a sudden increase in the bit rate. This rate penalty can have various causes. For example, an IRAP picture is much slower than a non-IRAP picture. are represented by significantly more bits than the inter-predicted pictures used as vectors. Furthermore, the existence of an IRAP picture is indicative of inter prediction. In particular, IRAP pictures do not use the previous reference picture from the DPB. Refresh the decoding process by removing the previous reference picture. This means that the IRAP picture is a reference for use in coding pictures that follow the IRAP picture in decoding order. This reduces the availability of pictures and therefore the efficiency of the process.

[0094] IDR pictures have a different signaling and derivation process than other IRAP picture types. For example, the signaling and derivation process associated with the IDR may be Instead of deriving the most significant bit (MSB) from the previous key picture, the MSB part of the POC is set to 0. In addition, the slice header of an IDR picture may include a However, it does not have to include information used for other purposes such as CRA, trailing, TSA, etc. The reference picture type can be used to perform the reference picture marking process. Reference pictures, such as a reference picture set (RPS) or a reference picture list The reference picture marking process may include reference picture information in the DPB. Whether the status is used for reference or not used for reference The presence of an IDR means that the decryption process simply determines whether all references in the DPB are This indicates that the picture is to be marked as not to be used for reference, so the IDR picture For a given architecture, such information does not need to be signaled.

[0095] In addition to the picture type, the picture identification information by POC is also used for reference in inter prediction. For managing the use of reference pictures, for outputting pictures from the DPB, and for scaling motion vectors. It is used for multiple purposes, such as filtering, weighted prediction, etc. In some video coding systems, pictures in the DPB are reserved for short-term reference. may be used for long-term reference or may not be used for reference. A picture may be marked as not to be used for reference. When such a picture is no longer available, it can no longer be used for prediction. Pictures may be removed from the DPB when they are not needed for power. In a coding system, reference pictures are marked as short-term and long-term. A reference picture is considered to be a reference picture when the picture is no longer needed for prediction reference. Between these statuses, The conversion may be controlled by a marking process of the decoded reference pictures. Sliding window processes and / or explicit memory management control actions (MMCOs) ) process may be used as a marking mechanism for decoded reference pictures. The sliding window process ensures that the number of reference frames in the SPS is max_num_ref_frame When the number of short-term reference pictures is equal to the specified maximum number, denoted as s, the short-term reference picture is used for reference. A short-term reference picture is a picture that is not used for a long-term The MMCO process may be stored in the DPB on a first-in, first-out basis. An MMCO command may contain multiple MMCO commands. An MMCO command may consist of one or more short-term or may mark long-term reference pictures as not being used for reference, All pictures may be marked as unused for reference or may be mark the reference picture of or an existing short-term reference picture as long-term, and then The long-term reference picture may be assigned a long-term picture index.

[0096] In some video coding systems, reference picture marking operations and The process for outputting and deleting pictures from the DPB is performed after the picture is decoded. Other video coding systems use the RPS for reference picture management. The most drastic difference between the RPS mechanism and the MMCO / sliding window process is The main difference is that for each particular slice, the RPS is The goal is to provide a complete set of reference pictures to be used by subsequent pictures. Therefore, all pictures to be held in the DPB for use by current or future pictures are The complete set of architectures is signaled in the RPS. This is relative to the DPB. Unlike the MMCO / sliding window method, where only changes are signaled. According to the algorithm, to maintain the correct status of reference pictures in the DPB, No information from the previous picture is required. The picture decoding order and the operation of the DPB are determined by the RPS In some video coding systems, the In some video coding systems, picture marking and The buffer operations, including both output and deletion of decoded pictures from the DPB, are In other video coding systems, the quantization may be applied after the texture is decoded. First, the RPS is decoded from the slice header of the current picture, and then the RPS is decoded from the slice header of the current picture. The marking and buffering operations of may be applied before decoding the current picture. .

[0097] In VVC, the management of reference pictures may be summarized as follows: Two reference picture lists, denoted List 1 and List 2, are directly signaled and derived. They are based on the RPS or sliding window + MMCO process discussed above. The marking of a reference picture is done for both active and non-active entries in the reference picture list. Based directly on the active entry and on the reference picture list 0 and 1, Only active entries are used as reference indexes for inter prediction of CTUs. The information for the derivation of the two reference picture lists is the SPS, the PPS, and the slice. The information is signaled by syntax elements and syntax structures in the address header. A predefined RPL structure is provided for use by referencing it in the slice header. The two reference picture lists are signaled in the bidirectional inter-prediction (B) slice. All slices, including unidirectional inter-predicted (P) slices and intra-predicted (I) slices The two reference picture lists are generated for the slices of the same picture group. It may be constructed without the use of an initialization process or a reference picture list modification process. The long-term reference picture (LTRP) is identified by the POC LSB. Delta POC MSB cycles (d elta POC MSB cycle) is signaled with respect to the LTRP as determined per picture. This is also fine.

[0098] To code a video image, the image is first partitioned and the partitions are converted into a bitstream. Various picture partitioning schemes are available. The image is divided into regular slices, dependent slices, and tiles. , and / or Wavefront Parallel Processing (WPP). For simplicity, HEVC is In this case, the slices are divided into groups of CTBs for video coding. Only slices, subordinate slices, tiles, WPPs, and combinations of these may be used. Such partitioning is done by matching the size of the Maximum Transmission Unit (MTU). may be applied to support parallel processing, and reduced end-to-end delay. The MTU represents the maximum amount of data that can be sent in a single packet. If a packet exceeds the MTU, the payload is fragmented in a process called fragmentation. The data is split into two packets by the

[0099] Regular slicing, also known simply as slicing, is caused by loop filtering operations. with other normal slices in the same picture, even though there are some interdependencies are partitioned portions of an image that can be reconstructed independently. Each regular slice is , which are then encapsulated in proprietary Network Abstraction Layer (NAL) units for transmission. In addition, intra-picture prediction across slice boundaries (intra-sample prediction, motion information prediction, The interdependence of coding mode prediction and entropy coding is independent of each other. Such independent reconstructions may be disabled in parallel to support For example, regular slice-based parallelism supports minimal inter-processor parallelism. Or use inter-core communication. But since each normal slice is independent, Each slice is associated with a separate slice header. is due to the bit cost of the slice header for each slice and the slice boundaries. The lack of straddling prediction can lead to significant coding overhead. ,Normal slices are used to support matching on MTU size requirements. In particular, regular slices may be encapsulated in separate NAL units and When a regular slice is coded, the slice is divided into multiple packets. It should be smaller than the MTU of the MTU scheme to avoid fragmentation. For the purpose of parallelization and MTU size matching, the layout of slices in a picture is Conflicting requirements may be imposed on the outs.

[0100] A dependent slice is similar to a normal slice, but has an abbreviated slice header and This allows for the classification of image tree block boundaries without compromising intra-cloud prediction. Therefore, a dependent slice is a normal slice fragmented into multiple NAL units. This allows a portion of a normal slice to be encoded before the entire normal slice is completely coded. Reduced end-to-end delay by allowing packets to be sent before they are completed results.

[0101] A picture may be divided into tile groups / slices and tiles. A tile is A sequence of CTUs that cover a rectangular area of ​​the picture. Tile Group / Slice contains several tiles of a picture. Raster scan tile group mode and long The rectangular tile group mode may be used to generate the tiles. In the Contour Tile Group mode, the tile group is a raster scan of the tiles in the picture. In rectangular tile group mode, the tile group A pixel contains several tiles of a picture that collectively form a rectangular region of the picture. Tiles within a rectangular tile group are in raster scan order for the tile group. For example, a tile has horizontal and vertical boundaries that create columns and rows of tiles. A tile may be a segmented portion of an image generated by a raster scan. The scan order of the CTB may be coded in time sequence (right to left and top to bottom). Therefore, the CTB in the first tile must be restarted before proceeding to the CTB in the next tile. Like normal slices, tiles are coded in sparse scan order. This breaks the prediction interdependence and the entropy decoding interdependence. However, the tiles are The tiles may not be included in individual NAL units, so they may be sized to match the MTU. Each tile is processed by one processor / core. and can be used for intra-picture prediction between processing units that decode neighboring tiles. Inter-processor / inter-core communication used is shared (when neighboring tiles are in the same slice). Carrying the slice headers included in the sample and metadata The sharing of two or more tiles may be limited to performing group filtering. When a slice is included, each tag is offset except for the entry point offset at the beginning of the slice. The byte offset of the entry point for the file is signaled in the slice header. For each slice and tile, the following conditions may be satisfied: 1) Within a slice 1) All coded tree blocks of a tile belong to the same tile, and 2) All coded treeblocks in a slice belong to the same slice. At least one should be satisfied.

[0102] In WPP, the image is partitioned into a single row of CTBs. Entropy decoding and prediction mechanism The mechanism may use data from the CTB in other rows. This allows for parallel processing. For example, the current row may be decoded in parallel with the previous row. However, the decoding of the current row is delayed by 2CTB from the decoding process of the previous row. The extension is done by coding the CTB above the current CTB in the current line and the CTB above the current CTB in the upper right before the current CTB is coded. This technique ensures that relevant data for the CTB is available. When rendered, it appears as a wavefront. This offset onset is at most the same as the number of CTB rows that the image contains. It allows parallelization up to the same number of processors / cores. Since intra-picture prediction between rows is allowed, a processor for enabling intra-picture prediction is required. Inter-server / inter-core communication can be quite high. WPP division takes into account the size of NAL units. Therefore, WPP does not support MTU size matching. To enforce MTU size matching, normal slices are It can be used in conjunction with WPP with overhead. Finally, wavefront segments (wavefronts) An ont segment may contain exactly one CTB row. Furthermore, when using WPP and When a slice starts within a CTB row, the slice should end within the same CTB row.

[0103] The tiles may also include a motion constrained tile set. A motion constrained tile set (MCTS) is a set of tiles whose associated motion vectors are located at full-sample intervals in the MCTS. fractional sample locations, which require only full sample locations in the MCTS for interpolation. A tileset designed to be constrained to point to fractional samples. In addition, the motion vectors for temporal motion vector prediction derived from blocks outside the MCTS are In this way, each MCTS is not allowed to use tiles that are not included in the MCTS. The temporal MCTS supplemental enhancement information (SEI) may be decoded independently without the presence of The "cluster information" message indicates the presence of MCTS in the bitstream and synchronizes the MCTS. The MCTS SEI message may be used to signal the match for the MCTS set. (specified as part of the semantics of the SEI message) to generate a bitstream that Provides supplemental information that can be used in the extraction of MCTS sub-bitstreams (specified in the The information defines several sets of MCTSs, each of which is used for the MCTS sub-bitstream extraction process. Alternative Video Parameter Sets (VPS), Sequence Parameter Sets (SPS), and the Picture Parameter Set (PPS) Raw Byte Sequence Payload (RBSP) bytes The MCTS sub-bitstream extraction process extracts several sets of extracted information, including When extracting the sub-bitstreams using the may be rewritten or replaced, the slice header may be (first_slice_segment Syntax related to slice addresses (including nt_in_pic_flag and slice_segment_address) One or all of the sub-bitstreams may differ in the extracted sub-bitstream. The value may be used and therefore updated.

[0104] VR applications, also known as 360-degree video applications, are applications that display a portion of a complete sphere. A user may only view the entire picture and, as a result, only a subset of the entire picture may be displayed. Dynamic Adaptive Streaming over Transport Protocol (DASH) A mechanism for viewport-dependent 360 delivery via the HyperText Transfer Protocol (HyperText Transfer Protocol) Lowering the bit rate and supporting the delivery of 360-degree video via streaming mechanisms This mechanism may be used, for example, to Split the sphere / projected picture into multiple MCTSs by using bemap projection. The two or more bitstreams may be encoded at different spatial resolutions or qualities. When delivering data to the decoder, the M from the higher resolution / quality bitstream is used. CTS is sent for the viewport that is to be displayed (for example, the front viewport). MCTS from lower resolution / quality bitstreams are sent for other viewports. These MCTS are packed in a specific way and then received to be decoded. The viewport seen by the user creates a positive viewing experience. It is expected that the MCTS will be represented in high resolution / quality to allow the user to see the When you turn to look at a viewport (for example, the left or right viewport), the display The system fetches high resolution / quality MCTS for the new viewport. The image comes from a lower resolution / quality viewport for a short period of time. When the user turns their head to look at the viewport, There is a delay between when the higher resolution / quality representation of the port is seen. This delay is How quickly the system can fetch higher resolution / quality MCTS for that viewport. How quickly it can be fetched depends on the IR It depends on the period of the IRAP. The period of the IRAP is the interval between the occurrence of two IRAPs. This delay is Since the MCTS of the new viewport can only be decoded from the IRAP picture, Related to.

[0105] For example, if the period of the IRAP is coded as every second, then the following applies: The best case scenario for delay is when the system starts fetching new segments / IRAPs. The network when the user turns their head to see a new viewport just before The round trip delay is the same as the round trip delay. In this scenario, the system A higher resolution / quality MCTS can be requested immediately, thus minimizing the buffer size. The matching delay can be set to almost zero, and the sensor delay is small and negligible. Assuming that the only delay is the network round trip delay, is the delay of the fetch request plus the transmission time of the requested MCTS. The round trip delay can be, for example, about 200 milliseconds. The worst case scenario for the delay is ,The user's head moves to a new viewport after the system has already made a request for the next segment. The IRAP period when turning around to view + network round trip delay. The team will be more frequent so that the IRAP period is shorter to improve the worst case scenario above. The IRAP pictures can be coded using the IRAP picture scheme, which reduces the overall delay. However, this approach has a higher bandwidth requirement due to the reduced compression efficiency. become.

[0106] In an exemplary implementation, sub-pictures of the same coded picture are coded in different This mechanism is explained as follows: A picture may be divided into subpictures. A subpicture is a set of tile_ A rectangular set of tile groups / slice starting from the tile group with group_address Each sub-picture may reference a corresponding PPS and thus is a separate tile. The presence of sub-pictures may be indicated in the PPS. Subpictures are treated like pictures in the decoding process. In-loop filtering may always be disabled. The width and height of a subpicture are The position of the subpicture within the picture may be specified in units of luma CTU size. The subpictures may not be signaled, but may be derived using the following rules: A picture contains a CTU within a picture that is large enough to contain the subpicture within the picture boundary. Take the next unoccupied position in raster scan order of each subpicture. The reference picture is the current subpicture and the corresponding reference picture in the decoded picture buffer. It is generated by extracting the area to be located. The extracted area is decoded. subpictures that are aligned in the same direction and therefore have the same size and position within the picture. In such a case, inter prediction is performed between the frames. Allowing different nal_unit_type values ​​allows subpictures derived from random access pictures to be A picture can be created without much difficulty (e.g., a random access picture can be created with a subpicture derived from a random access picture). can be merged into the same coded picture (e.g. without any VCL level modifications) Such advantages also apply to MCTS-based coding.

[0107] Allowing different values ​​of nal_unit_type within a coded picture is not supported by other systems. For example, a user may view a portion of 360-degree video content in a Some areas may be viewed more frequently than others. MCTS / Subpicture based views Coding efficiency and average equivalent quality for 360° video delivery depending on the port To make a better tradeoff between viewport switching latency and other More frequent IRAP pictures are coded for areas that are more commonly seen than for other areas. Comparable quality viewport switching latency can be achieved by switching from the first viewport to the second. When switching to a different viewport, the presentation quality of the second viewport is different from that of the first viewport. is the latency experienced by the user until a presentation quality equivalent to that of the original is reached.

[0108] Another implementation is for mixed NALs within a picture, including derivation of POC and management of reference pictures. Use the following solution for unit type support: Mixed IRAP subpictures to specify whether pictures with IRAP and non-IRAP subpictures may be present. , Flags referenced directly or indirectly by tile groups (sps_mixed_tile_gr oups_in_pic_flag) is present in the parameter set. For units, whether the POC MSB is reset or not when the POC is derived for a picture is specified. To specify this, a flag (poc_msb_reset_flag) must be present in the corresponding tile group header. A variable called PicRefreshFlag is defined and associated with the picture. Lag: The derivation of POC and the state of DPB should be refreshed when decoding a picture. The value of PicRefreshFlag is derived as follows: If the image group is included in the first access unit of the bitstream, PicRefreshF The lag is set equal to 1. Otherwise, the current tile group is the IDR tile. If it is a group, PicRefreshFlag is sps_mixed_tile_groups_in_pic_flag ? poc_ms b_reset_flag : Set to 1. Otherwise, the current tile group is reset to CR If it is an A-tile group, the following applies: The current access unit is coded If this is the first access unit in a sequence being refreshed, PicRefreshFlag is equal to 1. The access unit is set to end of sequence NAL unit. The HandleCraAsFirstPicInCvsFlag variable that immediately follows or is associated with the When set to , the current access unit is the first access unit. Otherwise, PicRefreshFlag is set equal to 0. (e.g., the current tile group is in the first access unit of the bitstream. Not part of the IRAP tile group isn't it).

[0109] When PicRefreshFlag is equal to 1, the value of POC MSB (PicOrderCntMsb) is It is reset to be equal to 0 during the derivation of the POC. The information used for reference picture management, such as the reference picture list (RPL), is stored in the corresponding NAL unit. The reference pixel is signaled in the tile group / slice header regardless of the tile type. The feature list is built at the beginning of decoding of each tile group, regardless of the NAL unit type. The reference picture list is RefPicList

[0000] and RefPicList

[0001] for the RPL method, RefPicList0[ ] and RefPicList1[ ] for the RPS method, or The PicRefreshFl may contain a similar list containing reference pictures for center prediction operations. When ag is equal to 1, all reference pictures in the DPB are marked during the reference picture marking process. The structure is marked as unused for reference purposes.

[0110] Such implementations are associated with specific problems. For example, When mixing of type values ​​is not allowed and the derivation of whether a picture is an IRAP picture When the output and derivation of the variable NoRaslOutputFlag are described at the picture level, the decoder These derivations may be performed after receiving the first VCL NAL unit of any picture. However, due to the support for mixed NAL unit types within a picture, the decoder It must wait for the arrival of another VCL NAL unit before performing the derivation. If so, the decoder must wait for the arrival of the last VCL NAL unit of the picture. Furthermore, such a system requires that the POC MSB is reset during derivation of the POC for a picture. A flag is added in the tile group header of the IDR NAL unit to specify whether or not This mechanism has the following problems: For bit types and non-IRAP NAL unit types supported by this mechanism, Furthermore, this information is not encoded in the tile group / slice header of the VCL NAL unit. Signaling means that an IRAP (IDR or CRA) NAL unit is inserted into a picture along with a non-IRAP NAL unit. When there is a change in the status of whether the bitstream is mixed with the requires that the value be changed during the merge. Such modifications to the slice headers are It occurs whenever a user requests video and therefore consumes a lot of hardware resources. In addition, certain IRAP NAL unit types and certain non-IRAP NAL unit types require Some other mixture of different NAL unit types within a picture other than a mixture of bit types. Such flexibility may not provide support for practical use cases. On the other hand, they complicate the codec design, which unnecessarily increases the complexity of the decoder. , thus raising the associated implementation costs.

[0111] Generally, this disclosure relates to a method for implementing sub-picture or MCTS based runtime in video coding. More particularly, this disclosure describes techniques for supporting subpicture access. A picture in a frame that is used to support random access based on the MCTS or MCTS. This paper describes an improved design for mixed NAL unit type support. The description is based on the VVC standard, but also applies to other video / media codec specifications.

[0112] To solve the above problems, the following exemplary implementations are disclosed. In one example, each picture is applied to the other pictures. This is associated with an indication of whether the nal_unit_type value specified in This indication is signaled within the PPS. Marking the picture as not used for reference removes the POC MSB. Supports the determination of whether to reset and / or whether to reset the DPB. When an indication is signaled in a PPS, a change in the value in the PPS is may be performed during a separate extraction. However, this does not include the extraction of such a bitstream. or during a merger when the PPS is overwritten and replaced by other mechanisms It is acceptable.

[0113] Alternatively, this indication may be signaled in the tile group header. may be required to be the same for all tile groups of a picture. In this case, the value is Alternatively, this indication may need to be changed to The tile group size is signaled in the frame header but is the same for all tile groups of a picture. However, in this case the value must be the MCTS / subpicture sequence. It may be necessary to modify the bitstream during extraction of the sub-bitstream of the data. When the indication is used for a picture, all VCL NALs of the picture Define additional VCL NAL unit types whose units have the same NAL unit type value. However, in this case, the Nth part of the VCL NAL unit The value of the AL unit type is used for the extraction of sub-bitstreams of MCTS / Subpicture sequences. This indication may need to be changed during the When all VCL NAL units of a picture are used for a By defining additional IRAP VCL NAL unit types that have the following values: However, in this case, the value of the NAL unit type of the VCL NAL unit is Must be modified during extraction of sub-bitstreams of MCTS / Subpicture sequences. Alternatively, at least one VCL N of any of the IRAP NAL unit types may be Each picture with an NAL unit contains a value of the NAL unit type in which the picture is mixed. The suffix may be associated with an indication of whether the

[0114] Furthermore, only mixed IRAP and non-IRAP NAL unit types are allowed. This allows mixing of nal_unit_type values ​​within a picture in a limited way. Such constraints may be applied: For any particular picture, all VCL NAL units Either some VCL NAL units have the same NAL unit type or some VCL NAL units have a specific IRAP N AL unit type and the rest have specific non-IRAP VCL NAL unit types. In other words, the VCL NAL units of any particular picture may be split into two or more IRAP NAL unit types, and may have more than one non-IRAP NAL unit type. A picture does not contain a mixed picture nal_unit_type value and is not a VCL NAL A unit may be considered an IRAP picture only if it has an IRAP NAL unit type. For all IRAP NAL units that do not belong to an IRAP picture (including IDR), the POC MSB is All IRAP NAL units (including IDR) that do not belong to an IRAP picture are reset. For units, the DPB is not reset, and therefore all reference pictures are Marking the TemporalId as unused for If at least one VCL NAL unit of a picture is an IRAP NAL unit, may be set equal to 0.

[0115] The following are specific implementations of one or more of the above-described aspects. If the value of u_types_in_pic_flag is equal to 0 and each VCL NAL unit has IDR_W_RADL and RSV_IRAP_ Code with nal_unit_type in the range from IDR_W_RADL to RSV_IRAP_VCL13, including VCL13 An exemplary PPS syntax and section The mantics are as follows: [Table 2] mixed_nalu_types_in_pic_flag specifies that each picture that references a PPS contains multiple VCL NAL units. and to specify that these NAL units do not have the same value of nal_unit_type mixed_nalu_types_in_pic_flag is set equal to 0 for each picture that references a PPS. Equal to 0 to specify that the VCL NAL units of this It is set as follows.

[0116] An example tile group / slice header syntax is as follows: [Table 3A] [Table 3B]

[0117] The semantics of an exemplary NAL unit header are as follows: For VCL NAL units in a frame, one of the following two conditions is met: All VCL NAL units have the same value of nal_unit_type. , specific IRAP NAL unit type values ​​(i.e., IDR_W_RADL and RSV_IRAP_VCL13) While all have a value of nal_unit_type in the range IDR_W_RADL to RSV_IRAP_VCL13, All other VCL NAL units are encoded as specific non-IRAP VCL NAL unit types (i.e., TRAIL_ Within the range from TRAIL_NUT to RSV_VCL_7, including NUT and RSV_VCL_7, or within the range from RSV_VCL14 and and RSV_VCL15, inclusive) The value of nuh_temporal_id_plus1 minus 1 is the temporal identifier for the NAL unit. al identifier). The value of nuh_temporal_id_plus1 is not equal to 0.

[0118] The variable TemporalId is derived as follows: TemporalId = nuh_temporal_id_plus1 - 1 (7-1)

[0119] IDR_W_RADL and RSV_IRAP_VCL13 for VCL NAL units with nal_unit_type of picture When the VC is within the range from IDR_W_RADL to RSV_IRAP_VCL13, including Regardless of the value of the nal_unit_type of the NAL unit, the TemporalId is the same for all VCLs of a picture. For a NAL unit, it is equal to 0. The value of TemporalId is the same for all VCL NAs of an access unit. The same applies for the L unit. The value of emporalId is the VCL NAL unit of a coded picture or access unit. The TemporalId value of the

[0120] An exemplary decoding process for a coded picture is as follows: The decoding process operates as follows for the current picture, CurrPic: The following decoding process is performed based on the tile group header It uses syntax elements from the layer and above. The variables and functions related to the count are derived as detailed herein. This is only called for the first tile group / slice of a picture. At the beginning of the decoding process for a picture group / slice, The decoding process for this purpose uses reference picture list 0 (RefPicList

[0000] ) and reference picture list It is called to derive RefPicList

[0001] . The current picture is an IDR picture. In this case, the decoding process for constructing the reference picture list is It may be called to check conformance, but not for the current picture or for the current picture in decoding order. It may not be necessary for the decoding of pictures that follow the picture.

[0121] The decoding process for constructing the reference picture list is as follows: is called at the beginning of the decoding process for each tile group. A reference index is a list of reference pictures. When decoding an I-tile group, the reference picture list is It is not used in decoding P tile group data. Only reference picture list 0 (RefPicList

[0000] ) is used in decoding tile group data. When decoding a B tile group, reference picture list 0 and reference picture list Both RefPicList

[0001] and RefPicList

[0002] are used in decoding the tile group data. At the beginning of the decoding process for a tile group, the reference picture list RefPicList

[0000] and The reference picture list is derived by marking the reference pictures. It is used in the decoding of IDR picture data or in the decoding of tile group data. For all tile groups or I-tile groups of non-IDR pictures, RefPicList

[0000] and RefPicList

[0001] may be derived for the purpose of checking bitstream conformance. However, their derivation depends on the current picture or the picture that follows it in decoding order. For P tile groups, RefPicList

[0001] is the bitstream. It may be derived for the purpose of checking the conformance of the stream, but the derivation is based on the current picture or It is not necessary for the decoding of pictures that follow the current picture in decoding order.

[0122] 8 is a schematic diagram of an example video coding device 800. The device 800 is adapted to implement the disclosed examples / embodiments as described herein. The video coding device 800 is suitable for use in a downstream port 820, an upstream port 830, and a downstream port 840. Stream ports 850 and / or upstream and / or downstream through the network A transceiver unit (including a transmitter and / or a receiver) for transmitting data in a stream The video coding device 800 includes a logical unit for processing data. A processor 830 including a memory and / or a central processing unit (CPU), and a The video coding device 800 may further include an electrical, optical, or wireless Upstream ports 850 and 851 for data transmission over wireless communication networks and / or electrical, optical-electrical (OE) components, electrical- It may also include optical (EO) components, and / or wireless communication components. The input device 800 includes an input and / or output unit for communicating data to and from a user. The system may also include an input / output (I / O) device 860. The I / O device 860 may display video data. output devices such as a display for displaying the image data and a speaker for outputting audio data. The I / O device 860 may include input devices such as a keyboard, a mouse, and a trackball. and / or corresponding interfaces for interacting with such output devices. It may also include an interface.

[0123] The processor 830 is implemented in hardware and software. 30 refers to one or more CPU chips, cores (e.g., as a multi-core processor), field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), and Digital Signal The processor 830 may be implemented as a digital signal processor (DSP). 0, Tx / Rx 810, upstream port 850, and memory 832. 0 includes a coding module 814. The coding module 814 includes a CVS 500, a VR pin The method may use the texture video stream 600 and / or the bitstream 700. The methods for implementing the disclosed embodiments described herein, such as methods 100, 900, and 1000, are The coding module 814 may use any other method / media described herein. Additionally, the coding module 814 may implement a codec system. 200, the encoder 300, and / or the decoder 400. The encoding module 814 encodes a picture containing both IRAP NAL units and non-IRAP NAL units. It indicates when it contains a single type of IRAP NAL unit and a single type of non-IRAP NAL unit. A flag may be set in the PPS to constrain such pictures to contain only Therefore, the coding module 814, when coding the video data, 8. Provides video coding device 800 with additional functionality and / or coding efficiency. Thus, coding module 814 enhances the functionality of video coding device 800. It improves the performance and addresses the problems inherent in video coding technology. Module 814 effects the transition of video coding device 800 into a different state. Generally, the coding module 814 is stored in a memory 832 and executed by a processor 830. as instructions to be executed (e.g., a computer program product stored on a non-transitory medium) It can be implemented as

[0124] Memory 832 can be a disk, tape drive, solid state drive, read-only Memory (ROM), Random Access Memory (RAM), Flash Memory, Ternary Content Addressable Memory (TCAM: ternary content-addressable memory (CRAM), static random access memory (SRAM), etc. The memory 832 may include one or more memory devices that are configured to execute a program when the program is selected for execution. for storing such a program in a memory and for storing instructions and instructions to be read during execution of the program and an overflow data storage device for storing data. It may also be used as an orage device.

[0125] FIG. 9 shows a composite video signal generated by merging multiple sub-picture video streams 601 to 603 with multiple video resolutions. A bitstream such as bitstream 700 containing the VR picture video stream 600 A video sequence such as CVS 500 that contains pictures with mixed NAL unit types 9 is a flow diagram of an example method 900 for encoding a signal. In this case, the codec system 200, the encoder 300, and / or the video coding device This may be used by an encoder such as encoder 800.

[0126] The method 900 includes a method for encoding a video sequence including a plurality of pictures, such as VR pictures, into a video stream. and encoding the video sequence into a bitstream based on, for example, user input. In step 901, the current picture is coded. The encoder determines whether the image contains multiple sub-pictures of different types. The type must contain at least one slice of a picture that contains part of an IRAP subpicture and a non-IRA A P NAL sub-picture may include at least one slice of a picture that includes a portion of the P NAL sub-picture. In step 903, the encoder converts the slices of the subpictures of the picture into a bitstream. Such a VCL NAL unit may consist of one or more A packet may contain IRAP NAL units and one or more non-IRAP NAL units. The encoding step converts the sub-bitstreams of different resolutions into a single bitstream for transmission to a decoder. This may include merging the bitstream into a

[0127] In step 905, an encoder encodes the PPS into a bitstream and sets the flag to As a specific example, encoding a PPS may be done using For example, depending on the merging of the sub-bitstreams, the already coded PPS may be modified to include the flag value. The flag may include changing the value of the NAL unit type associated with the picture. When the VCL NAL unit is the same for all VCL NAL units in a The flag also relates to a VCL NAL unit that contains one or more of the picture's subpictures. The first NAL unit type value is a VCL NAL unit that contains one or more of the subpictures of a picture. When the second NAL unit type value for the unit is different from the second NAL unit type value for the unit, the second NAL unit type value may be set to a second value. For example, the first NAL unit type value indicates that the picture contains an IRAP subpicture. The second NAL unit type value may indicate that the picture also contains non-IRAP subpictures. Furthermore, the value of the first NAL unit type may indicate IDR_W_RADL, IDR_N_LP, or In addition, the value of the second NAL unit type may be equal to one of TR_NUT or CRA_NUT. AIL_NUT, RADL_NUT, or RASL_NUT. The flag may be mixed_nalu_types_in_pic_flag. _types_in_pic_flag indicates that each picture that references the PPS that contains the flag contains two or more VCL NAL units. In addition, the flag may be set equal to 1 to specify that the The tag indicates that all VCL NAL units associated with the corresponding picture are of the NAL unit type (nal_uni t_type) in a mixed_nal u_types_in_pic_flag indicates that each picture that references the PPS containing the flag is a VCL NAL unit. In addition, the flag ensures that all VCL NAL units of the corresponding picture have the same value of nal_unit_type Specify.

[0128] In step 907, the encoder generates a bitstream for transmission to the decoder. It may be memorized.

[0129] FIG. 10 shows a composite video signal merging multiple sub-picture video streams 601 to 603 with multiple video resolutions. A bitstream such as bitstream 700 containing the VR picture video stream 600 A video sequence such as CVS 500 that contains pictures with mixed NAL unit types from different streams 1 is a flow diagram of an example method 1000 for decoding an instance of a digital signal. Sometimes the codec system 200, the decoder 400, and / or the video coding device This may be used by a decoder such as the processor 800.

[0130] The method 1000 may, for example, be implemented by generating a coded image representing a video sequence as a result of the method 900. Step 1 may begin when the decoder begins to receive a bitstream of encoded data. In step 1001, a decoder receives a bitstream. In a particular example, the bitstream includes a number of subpictures and flags associated with the subpicture. A stream may contain a PPS that contains a flag. In addition, a subpicture may be split into multiple VCL NAL units. For example, a slice associated with a subpicture is included in a VCL NAL unit. Included.

[0131] In step 1003, when the flag is set to a first value, the decoder The bit type value is determined to be the same for all VCL NAL units associated with a picture. Furthermore, when the flag is set to a second value, the decoder The first NAL unit type value for a VCL NAL unit that contains one or more pictures is The second NAL unit for a VCL NAL unit that contains one or more of the subpictures of a picture. For example, the value of the first NAL unit type is different from the value of the picture The second NAL unit type value may indicate that the NAL unit contains an IRAP subpicture, and the second NAL unit type value may indicate that the picture In addition, the first NAL unit type may indicate that the image also contains non-IRAP subpictures. The value may be equal to one of IDR_W_RADL, IDR_N_LP, or CRA_NUT. The value of NAL unit type 2 is equal to one of TRAIL_NUT, RADL_NUT, or RASL_NUT. As a specific example, the flag may be mixed_nalu_types_in_pic_flag. mixed_nalu_types_in_pic_flag specifies that each picture that references a PPS contains two or more VCL NAL units. and the VCL NAL units do not have the same value of NAL unit type (nal_unit_type). When specifying that mixed_nalu_types_in_p ic_flag indicates that each picture that references a PPS has one or more VCL NAL units that reference the PPS. equal to 0 when the VCL NAL units of each picture in the may be set to

[0132] In step 1005, the decoder determines the subpicture based on the value of the NAL unit type. The decoder may also decode one or more of the decoded Transfer one or more of the subpictures for display as part of the combined video sequence. You may do so.

[0133] FIG. 11 shows a composite video signal generated by merging multiple sub-picture video streams 601 to 603 with multiple video resolutions. A bitstream such as bitstream 700 containing the VR picture video stream 600 A video sequence such as CVS 500 that contains pictures with mixed NAL unit types 1 is a schematic diagram of an example system 1100 for coding a code. The decoder system 200, the encoder 300, the decoder 400, and / or the video coding The system may be implemented by an encoder and a decoder, such as the device 800. The system 1100 may be used when performing the methods 100, 900, and / or 1000.

[0134] The system 1100 includes a video encoder 1102. The video encoder 1102 is configured to receive a picture A determination module 1101 for determining whether the image includes multiple sub-pictures of different types. The video encoder 1102 may include a plurality of sub-pictures of a picture in a bitstream. The encoding module 1103 further includes a coding module 1103 for coding the VCL NAL unit of the encoded data. The decoding module 1103 decodes all VCL NAL units whose NAL unit type values ​​are related to pictures. It is set to the first value when the pixels are the same for all the units in the picture and one of the subpictures of the picture. The first NAL unit type value for a VCL NAL unit that contains one or more subpictures of a picture The second NAL unit type value for the VCL NAL unit that contains one or more of the and encoding into the bitstream a flag that is set to a second value when the The video encoder 1102 stores the bitstream for transmission to the decoder. The video encoder 1102 further includes a storage module 1105 for storing the bitstream. The video encoder further includes a transmission module 1107 for transmitting to a video decoder 1110. 1102 may be further configured to perform any of the steps of method 900.

[0135] The system 1100 also includes a video decoder 1110. The video decoder 1110 receives and stores picture-related A receiving module for receiving a bitstream including a plurality of sub-pictures and a flag for A subpicture, including module 1111, is contained in multiple VCL NAL units. When the flag is set to a first value, the NAL unit type value is set to a picture. a determination module 1113 for determining that the VCL NAL units are the same for all related VCL NAL units; Further, the determination module 1113 further includes, when the flag is set to a second value: The first NAL unit for a VCL NAL unit that contains one or more of the subpictures of a picture. The value of the type is for a VCL NAL unit that contains one or more of the subpictures of a picture. The video decoder 1110 determines that the value of the second NAL unit type is different from that of the first NAL unit type. , and a decoder for decoding one or more of the subpictures based on the value of the NAL unit type. The video decoder 1110 further includes a signaling module 1115 for decoding the decoded video sequence. a transfer module for transferring one or more of the subpictures for display as part of the The video decoder 1110 further includes a processor 1117. The video decoder 1110 may perform any of the steps of the method 1000. It may be further configured as follows.

[0136] The first component may include a line, trace, or other connection between the first component and the second component. When there are no intervening components other than the medium, the first component is directly coupled to the second component. An element is an intermediate entity other than a line, trace, or other medium between a first and a second entity. When there is an intermediate component, it is indirectly connected to the second component. and variations thereof include both directly and indirectly linked. The use of "+ / - 10%" means a range including ±10% of the number thereafter, unless otherwise stated. do.

[0137] The steps of the example methods described herein may not necessarily be performed in the order described. The order of steps in such methods is not required to be performed in any particular order, and the order of steps in such methods is exemplary only. It should also be understood that further steps may be taken to Various methods may be included, and certain steps may be included in methods consistent with various embodiments of the present disclosure. In some cases, they may be omitted or combined.

[0138] Although several embodiments have been provided in this disclosure, the disclosed systems and methods It should be understood that the present disclosure may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. It will be understood that these examples are intended to be illustrative and not limiting. The invention should be understood and the intention should not be limited to the details given in this specification. For example, various elements or components may be combined or integrated into another system. or certain features may be omitted or not implemented.

[0139] In addition, what is described in various embodiments as being separate or distinct may be The illustrated techniques, systems, subsystems, and methods are not intended to depart from the scope of the present disclosure. or combined or integrated with any other system, component, technique, or methodology. Other examples of modifications, substitutions, and alterations may occur to those skilled in the art. may be made without departing from the spirit and scope of the disclosure herein. It's possible. [Explanation of symbols]

[0140] 100 How it works 200 Encoding and Decoding (Codec) Systems 201 segmented video signal 211 General Coda Control Components 213 Transformation, Scaling and Quantization Components 215 Intra-picture Estimation Components 217 Intra-picture prediction components 219 Motion Compensation Components 221 Motion Estimation Components 223 Decoded Picture Buffer Components 225 In-loop filter components 227 Filter Control Analysis Components 229 Scaling and Inverse Transformation Components 231 Header Format and Context-Adaptive Binary Arithmetic Coding (CABAC) Configuration element 300 Video Encoder 301 Segmented Video Signal 313 Transformation and Quantization Components 317 Intra-picture prediction components 321 Motion Compensation Components 323 Decoded Picture Buffer Components 325 In-loop filter components 329 Inverse Transformation and Quantization Components 331 Entropy Coding Components 400 Video Decoder 417 Intra-picture prediction components 421 Motion Compensation Components 423 Decoded Picture Buffer Components 425 In-loop filter components 429 Inverse Transformation and Quantization Components 433 Entropy Decoding Components 500 CVS 502 IRAP Picture 504 Leading Picture 506 Trailing Picture 508 Decoding Order 510 Presentation order 600 VR Picture Video Streams 601 Subpicture Video Stream 602 Subpicture Video Stream 603 Subpicture Video Stream 700 bitstream 710 Sequence Parameter Set (SPS) 711 Picture Parameter Set (PPS) 715 slice header 720 Image data 721 Pictures 723 Subpicture 725 slices 727 mixed_nalu_types_in_pic_flag 730 Non-VCL NAL Units 731 SPS NAL unit type (SPS_NUT) 732 PPS NAL unit type (PPS_NUT) 740 VCL NAL units 741 IDR_N_LP NAL unit 742 IDR_w_RADL NAL unit 743 CRA_NUT 745 IRAP NAL Unit 746 RASL_NUT 747 RADL_NUT 748 TRAIL_NUT 749 Non-IRAP NAL Units 800 Video Coding Device 810 Transceiver Unit (Tx / Rx) 814 Coding Module 820 downstream ports 830 Processor 832 Memory 850 Upstream Ports 860 Input and / or Output (I / O) Devices 900 ways 1000 ways 1100 System 1101 Judgment module 1102 Video Encoder 1103 Encoding Module 1105 Memory Module 1107 Transmitting Module 1110 Video Decoder 1111 Receiver Module 1113 Judgment module 1115 Decryption Module 1117 Transfer Module

Claims

1. A method implemented in a decoder, comprising the steps of: A bitstream including a plurality of sub-pictures and flags associated with a picture is receiving by a receiver of a video coder, the sub-picture being a video coding layer (VCL) network abstraction layer (NAL) unit; When the flag is set to a first value, a value of a first NAL unit type is set to the picture. the processor determines that the VCL NAL units associated with the VCL NAL units are the same for all of the VCL NAL units associated with the VCL NAL units. A determining step; When the flag is set to a second value, The value of the first NAL unit type for the VCL NAL unit includes one or more of a second NAL unit for a VCL NAL unit including one or more of the subpictures of the image; determining by said processor that the value of said bit type is different from the value of said bit type; based on the value of the first NAL unit type or the value of the second NAL unit type and decoding, by the processor, one or more of the sub-pictures. method.

2. The bitstream includes a picture parameter set (PPS) including the flag. The method according to claim 1.

3. The value of the first NAL unit type indicates that the picture is an intra random access point. a value of the second NAL unit type indicating that the picture contains an Interleaved Random Access Point (IRAP) subpicture; 3. The method of claim 1 or 2, wherein the subpicture indicates that the image contains non-IRAP subpictures.

4. The value of the first NAL unit type is a random access decodable leading picture. Instantaneous Decoding Refresh (IDR) with leading picture (IDR_W_RADL), IDR without leading picture (IDR_N_LP), or the clean random access (CRA) NAL unit type (CRA_NUT). The method according to any one of claims 1 to 3.

5. The value of the second NAL unit type is a trailing picture NAL unit type (TRAIL_ NUT), random access decodable leading picture NAL unit type (RADL_NUT), or Random Access Skip Leading Picture (RASL) NAL unit type 5. The method according to claim 1, wherein the RASL_NUT is equal to RASL_NUT.

6. 6. The method according to claim 1, wherein the flag is mixed_nalu_types_in_pic_flag. Method of posting.

7. The picture that references the PPS has two or more of the VCL NAL units, and the VCL N To specify that AL units do not have the same value of NAL unit type (nal_unit_type) , mixed_nalu_types_in_pic_flag is equal to 1, and the picture that references the PPS is in the VC L NAL unit, and the VCL NAL unit has the same value of nal_unit_type. When specifying that the mixed_nalu_types_in_pic_flag is equal to 0, 2. The method according to any one of claims 1 to 11.

8. 1. A method implemented in an encoder, comprising: The processor determines whether a picture contains multiple subpictures of different types. determining a The sub-pictures of the picture are encoded into multiple video coding regions in a bitstream. The processor encodes the VCL into network abstraction layer (NAL) units. Steps and The first NAL unit type value is set for all VCL NAL units associated with the picture. and a first value when the first and second subpictures of the picture are the same as each other, The value of the first NAL unit type for the VCL NAL unit including A second NAL unit type for VCL NAL units that contain one or more of the subpictures. a flag set to a second value by said processor when said bitstream differs from said first value; encoding the data into a stream; A memory coupled to the processor for transmitting the bitstream to a decoder. and storing the information by the library.

9. encoding a picture parameter set (PPS) into the bitstream.

9. The method of claim 8, further comprising the step of: encoding the flag into the PPS. 。

10. The value of the first NAL unit type indicates that the picture is an intra random access point. a value of the second NAL unit type indicating that the picture contains an Interleaved Random Access Point (IRAP) subpicture; 10. The method of claim 8 or 9, which indicates that the image contains non-IRAP sub-pictures.

11. The value of the first NAL unit type is a random access decodable leading picture. Instantaneous Decoding Refresh (IDR) with leading picture (IDR_W_RADL), IDR without leading picture (IDR_N_LP), or the clean random access (CRA) NAL unit type (CRA_NUT). The method according to any one of claims 8 to 10.

12. The value of the second NAL unit type is a trailing picture NAL unit type (TRAIL_ NUT), random access decodable leading picture NAL unit type (RADL_NUT), or Random Access Skip Leading Picture (RASL) NAL unit type 12. The method according to claim 8, wherein the RASL_NUT is equal to .

13. 13. The method according to claim 8, wherein the flag is mixed_nalu_types_in_pic_flag. The method described.

14. The picture that references the PPS has two or more of the VCL NAL units, and the VCL N To specify that AL units do not have the same value of NAL unit type (nal_unit_type) , mixed_nalu_types_in_pic_flag is equal to 1, and the picture that references the PPS is in the VC L NAL unit, and the VCL NAL unit has the same value of nal_unit_type. When specifying that the mixed_nalu_types_in_pic_flag is equal to 0, 2. The method according to any one of claims 1 to 11.

15. A processor, a receiver coupled to the processor, and a memory coupled to the processor. a memory and a transmitter coupled to the processor, The memory and the transmitter are adapted to carry out the method according to any one of claims 1 to 14. A video coding device configured to:

16. A computer program product for use with a video coding device. A non-transitory computer-readable medium comprising a computer program product, The video coding device according to any one of claims 1 to 14, when executed by the processor. A computer program stored on a non-transitory computer readable medium for carrying out the method according to any one of claims 1 to 5. A non-transitory computer-readable medium containing computer-executable instructions.

17. Receiving a bitstream containing multiple subpictures and flags associated with a picture a receiving means for receiving the sub-pictures, the sub-pictures being encoded by a plurality of video coding layers (VCL ) a receiving means included in a network abstraction layer (NAL) unit; A determination means, When the flag is set to a first value, the value of the first NAL unit type is the same for all VCL NAL units associated with the chunk, When the flag is set to a second value, The value of the first NAL unit type for a VCL NAL unit including one or more of the above-mentioned pixels is a second NAL unit for a VCL NAL unit including one or more of the subpictures of the a determination means for determining that the value of the type is different from the value of the type of the based on the value of the first NAL unit type or the value of the second NAL unit type and decoding means for decoding one or more of the sub-pictures.

18. Claim 17, further configured to carry out the method according to any one of claims 1 to 7. A decoder according to claim 1.

19. Method for determining whether a picture contains multiple sub-pictures of different types Step by step, An encoding means, The sub-pictures of the picture are encoded as multiple video coding Layer (VCL) to Network Abstraction Layer (NAL) units, The first NAL unit type value is set for all VCL NAL units associated with the picture. is set to a first value when the two subpictures of the picture are the same; the value of the first NAL unit type for a VCL NAL unit that includes one or more a second NAL unit type for a VCL NAL unit that includes one or more of the subpictures; a flag set to a second value when different from the first value in the bitstream; and an encoding means for encoding the and storage means for storing said bitstream for transmission to a decoder. Encoder.

20. Claim 1 further configured to carry out the method according to any one of claims 8 to 14.

10. An encoder as recited in claim 9.

Images