Code stream decoding method, decoder and code stream storage equipment

Abstract

The invention discloses a video coding mechanism. In one embodiment, the mechanism includes: receiving a code stream, where the code stream includes a plurality of sub-images and a flag associated with an image, the sub-images being included in a plurality of video coding layer (VCL) network abstraction layer (NAL) units, and transmitting the code stream to the video coding layer (VCL) unit, where the sub-images are included in the plurality of VCL network abstraction layer (NAL) units, where the plurality of VCL network abstraction layer (NAL) units, where the plurality of VCL network abstraction layer (NAL) units, where the plurality of VCL network abstraction layer (NAL) units. When the flag is set to a first value, NAL unit type values of all VCL NAL units associated with the image are the same; when the flag is set to a second value, a first NAL unit type value including a VCL NAL unit of one or more sub-images in the image is different from a second NAL unit type value including a VCL NAL unit of one or more sub-images in the image; decoding the sub-image according to the NAL unit type value; the sub-images are forwarded for display as part of a decoded video sequence.

Description

[0001] Cross-reference to related applications

[0002] This application is a divisional application. The application number of the original application is 202080019614.8, and the original application date is March 11, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0003] The present invention relates generally to video coding, and more particularly to coding sub-pictures in pictures in video coding. Background Art

[0004] Even in the case of shorter videos, a large amount of video data is required to describe it, which can cause difficulties when the data is to be streamed or otherwise transmitted across a communication network with limited bandwidth capacity. Therefore, video data is often compressed before being sent across modern telecommunications networks. Since memory resources may be limited, the size of the video may also become an issue when storing the video on a storage device. Video compression devices typically use software and / or hardware on the source side to decode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination side by a video decompression device that decodes the video data. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the invention

[0005] In one embodiment, the present invention includes a method implemented in a decoder, the method including: a receiver of the decoder receives a code stream, the code stream including multiple sub-images and a flag associated with an image, wherein the sub-image is included in a video coding layer (VCL) network abstraction layer (NAL) unit; when the flag is set to a first value, a processor determines that the first NAL unit type value of all VCL NAL units associated with the image is the same; when the flag is set to a second value, the processor determines that the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image; the processor decodes the one or more sub-images according to the first NAL unit type value or the second NAL unit type value.

[0006] An image can be divided into multiple sub-images. These sub-images can be decoded into separate sub-streams, which can then be merged into a code stream to be sent to a decoder. For example, sub-images can be used in virtual reality (VR) applications. In a specific example, a user can view only a portion of a VR image at any time. Accordingly, different sub-images can be transmitted at different resolutions, so that more bandwidth can be allocated to sub-images that are likely to be displayed, and sub-images that are unlikely to be displayed can be compressed to improve decoding efficiency. In addition, a video stream can be encoded using intra-random access point (IRAP) images. IRAP images are decoded according to intra-frame prediction and can be decoded without reference to other images. Non-IRAP images can be decoded according to inter-frame prediction and can be decoded by reference to other images. Non-IRAP images are more compressed than IRAP images. However, the video sequence must start decoding with an IRAP image because the IRAP image includes enough data to be decoded without reference to other images. IRAP images can be used in sub-images to support dynamic resolution changes. Therefore, the video system can transmit more IRAP images for sub-images that are more likely to be viewed (for example, based on the user's current viewing angle) and transmit fewer IRAP images for sub-images that are less likely to be viewed, so as to further improve decoding efficiency. However, the sub-images are part of the same image. Therefore, the scheme can obtain an image that includes both IRAP sub-images and non-IRAP sub-images. Some video systems do not have the ability to process mixed images that have both IRAP and non-IRAP areas. The present invention includes a flag indicating whether the image is a mixed image, and therefore includes both IRAP components and non-IRAP components. Based on this flag, the decoder can process different sub-images in different ways when decoding so as to correctly decode and display the image / sub-image. The flag can be stored in the PPS and can be called mixed_nalu_types_in_pic_flag. Therefore, the disclosed mechanism can implement additional functions. In addition, the disclosed mechanism can realize dynamic resolution change when using sub-image code streams. Therefore, the disclosed mechanism can transmit low-resolution sub-image code streams when streaming VR video without significantly reducing the user experience. Therefore, the disclosed mechanism improves decoding efficiency, thereby reducing the usage of network resources, memory resources and processing resources at the encoder and the decoder.

[0007] Optionally, according to any one of the above aspects, in another implementation manner of the aspect, the code stream includes a picture parameter set (PPS), and the picture parameter set includes the flag.

[0008] Optionally, according to any one of the above aspects, in another implementation of the aspect, the first NAL unit type value indicates that the image includes an intra-random access point (IRAP) sub-image, and the second NAL unit type value indicates that the image includes a non-IRAP sub-image.

[0009] Optionally, according to any of the above aspects, in another implementation of the aspect, the first NAL unit type value is equal to an instantaneous decoding refresh (IDR) with a decodable random access preceding picture (IDR_W_RADL), an IDR without a preceding picture (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT).

[0010] Optionally, according to any of the above aspects, in another implementation of the aspect, the second NAL unit type value is equal to the trailing image NAL unit type (TRAIL_NUT), the decodable random access leading image NAL unit type (RADL_NUT), or the skipped random access leading image (random access skipped leading, RASL) NAL unit type (RASL_NUT).

[0011] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the flag is mixed_nalu_types_in_pic_flag.

[0012] Optionally, according to any of the above aspects, in another implementation of the aspect, when the image representing the reference to the PPS has multiple VCL NAL units and the VCL NAL units do not have the same NAL unit type (nal_unit_type) value, mixed_nalu_types_in_pic_flag is equal to 1; when the image representing the reference to the PPS has one or more VCL NAL units and the VCL NAL units have the same nal_unit_type value, mixed_nalu_types_in_pic_flag is equal to 0.

[0013] In one embodiment, the present invention includes a method implemented in an encoder, the method comprising: the processor determining whether an image includes multiple sub-images of different types; the processor encoding the sub-images in the image into multiple VCL NAL units in a codestream; the processor encoding a flag into the codestream, wherein when the first NAL unit type values ​​of all VCL NAL units associated with the image are the same, the flag is set to a first value, and when the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image, the flag is set to a second value; a memory coupled to the processor stores the codestream to be sent to a decoder.

[0014] An image can be divided into multiple sub-images. Such sub-images can be decoded into separate sub-streams, which can then be merged into a code stream to be sent to a decoder. For example, sub-images can be used in virtual reality (VR) applications. In a specific example, a user can view only a portion of a VR image at any time. Accordingly, different sub-images can be transmitted at different resolutions, so that more bandwidth can be allocated to sub-images that are likely to be displayed, and sub-images that are unlikely to be displayed can be compressed to improve decoding efficiency. In addition, a video stream can be encoded using intra-random access point (IRAP) images. IRAP images are decoded according to intra-frame prediction and can be decoded without reference to other images. Non-IRAP images can be decoded according to inter-frame prediction and can be decoded by reference to other images. Non-IRAP images are more compressed than IRAP images. However, the video sequence must start decoding with an IRAP image because the IRAP image includes enough data to be decoded without reference to other images. IRAP images can be used in sub-images and dynamic resolution changes can be achieved. Therefore, the video system can transmit more IRAP images for sub-images that are more likely to be viewed (for example, based on the user's current viewing angle) and fewer IRAP images for sub-images that are less likely to be viewed, so as to further improve decoding efficiency. However, the sub-images are part of the same image. Therefore, this scheme may produce images that include both IRAP sub-images and non-IRAP sub-images. Some video systems do not have the ability to process mixed images that have both IRAP and non-IRAP areas. The present invention includes a flag indicating whether the image is a mixed image, and therefore includes both IRAP components and non-IRAP components. Based on this flag, the decoder can process different sub-images in different ways when decoding in order to correctly decode and display the image / sub-image. The flag can be stored in the PPS and can be called mixed_nalu_types_in_pic_flag. Therefore, the disclosed mechanism can implement additional functionality. In addition, the disclosed mechanism can implement dynamic resolution changes when using sub-image code streams. Therefore, the disclosed mechanism can transmit low-resolution sub-image code streams when streaming VR video without significantly reducing the user experience. Therefore, the disclosed mechanism improves decoding efficiency, thereby reducing the usage of network resources, memory resources and processing resources at the encoder and the decoder.

[0015] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the method further includes encoding a PPS into the bitstream, wherein the flag is encoded into the PPS.

[0016] Optionally, according to any of the above aspects, in another implementation of the aspect, the first NAL unit type value indicates that the image includes an IRAP sub-image, and the second NAL unit type value indicates that the image includes a non-IRAP sub-image.

[0017] Optionally, according to any of the above aspects, in another implementation of the aspect, the first NAL unit type value is equal to IDR_W_RADL, IDR_N_LP or CRA_NUT.

[0018] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the second NAL unit type value is equal to TRAIL_NUT, RADL_NUT or RASL_NUT.

[0019] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the flag is mixed_nalu_types_in_pic_flag.

[0020] Optionally, according to any of the above aspects, in another implementation of the aspect, when the image representing the reference to the PPS has multiple VCL NAL units and the VCL NAL units do not have the same nal_unit_type value, mixed_nalu_types_in_pic_flag is equal to 1; when the image representing the reference to the PPS has one or more VCL NAL units and the VCL NAL units have the same nal_unit_type value, mixed_nalu_types_in_pic_flag is equal to 0.

[0021] In one embodiment, the present invention includes a video decoding device, comprising: a processor, a receiver coupled to the processor, and a memory coupled to the processor; a transmitter coupled to the processor, wherein the processor, receiver, memory and transmitter are used to execute the method described in any one of the above aspects.

[0022] In one embodiment, the present invention includes a non-transitory computer-readable medium, including a computer program product for use in a video decoding device, wherein the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium, and when a processor executes the computer-executable instructions, the video decoding device performs the method described in any of the above aspects.

[0023] In one embodiment, the present invention includes a decoder, the decoder including: a receiving module, used to receive a code stream, wherein the code stream includes multiple sub-images and a flag associated with an image, and the sub-images are included in multiple VCL NAL units; a determining module, used to: when the flag is set to a first value, determine that the first NAL unit type values ​​of all VCL NAL units associated with the image are the same; when the flag is set to a second value, determine that the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image; and a decoding module, used to decode the one or more sub-images according to the first NAL unit type value or the second NAL unit type value.

[0024] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the decoder is also used to execute the method described in any of the above aspects.

[0025] In one embodiment, the present invention includes an encoder, comprising: a determination module for determining whether an image includes multiple sub-images of different types; an encoding module for: encoding the sub-images in the image into multiple VCL NAL units in a code stream; encoding a flag into the code stream, wherein when the first NAL unit type values ​​of all VCL NAL units associated with the image are the same, the flag is set to a first value, and when the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image, the flag is set to a second value; and a storage module for storing the code stream to be sent to a decoder.

[0026] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the encoder is also used to execute the method described in any of the above aspects.

[0027] For clarity of description, any of the above-described embodiments may be combined with any or more of the other above-described embodiments to create new embodiments within the scope of the present invention.

[0028] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] For a more thorough understanding of the present invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0030] Figure 1 A flow chart of an exemplary method of decoding a video signal.

[0031] Figure 2 is a schematic diagram of an exemplary codec (codec) system for video coding.

[0032] Figure 3 is a schematic diagram of an exemplary video encoder.

[0033] Figure 4 is a schematic diagram of an exemplary video decoder.

[0034] Figure 5 is a schematic diagram of an exemplary decoded video sequence.

[0035] Figure 6 A schematic diagram of multiple sub-image video streams divided from a virtual reality (VR) image video stream.

[0036] Figure 7 is a schematic diagram of an exemplary code stream including an image with mixed Network Abstraction Layer (NAL) unit types.

[0037] Figure 8 is a schematic diagram of an exemplary video decoding device.

[0038] Fig. 9 A flow chart of an exemplary method for encoding a video sequence including pictures with mixed NAL unit types into a bitstream.

[0039] Fig.10 A flow chart of an exemplary method for decoding a video sequence including pictures with mixed NAL unit types from a codestream.

[0040] Fig.11 is a schematic diagram of an exemplary system for encoding a video sequence including pictures with mixed NAL unit types into a bitstream. DETAILED DESCRIPTION

[0041] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the systems and / or methods disclosed herein may be implemented using any number of techniques, whether currently known or existing. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.

[0042] The following abbreviations are used in this document: Coded Video Sequence (CVS), Decoded Picture Buffer (DPB), Instantaneous Decoding Refresh (IDR), Intra-Random Access Point (IRAP), Least Significant Bit (LSB), Most Significant Bit (MSB), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), and Working Draft (WD).

[0043] Many video compression techniques can be used to reduce video files while minimizing data loss. For example, video compression techniques can include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in a video sequence. For block-based video decoding, a video slice (e.g., a video image or a portion of a video image) can be divided into video blocks, which can also be referred to as treeblocks, coding tree blocks (CTB), coding tree units (CTU), coding units (CU) and / or coding nodes. Video blocks in an intra-frame decoding (I) slice of an image are decoded using spatial prediction of reference samples in adjacent blocks within the same image. Video blocks in an inter-frame decoding unidirectional prediction (P) or bidirectional prediction (B) slice of an image can be decoded using spatial prediction of reference samples in adjacent blocks within the same image or using temporal prediction of reference samples in other reference images. A picture (picture / image) can be referred to as a frame, and a reference picture can be referred to as a reference frame. Spatial or temporal prediction produces a prediction block representing an image block. The residual data represents the pixel difference between the original image block and the prediction block. Accordingly, the inter-frame decoding block is encoded according to the motion vector pointing to the block of reference samples constituting the prediction block and the residual data representing the difference between the coding block and the prediction block, while the intra-frame decoding block is encoded according to the intra-frame decoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to produce residual transform coefficients, which can then be quantized. The quantized transform coefficients can initially be arranged in a two-dimensional array. The quantized transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients. Entropy decoding can be applied to achieve a greater degree of compression. These video compression techniques will be discussed in more detail below.

[0044] In order to ensure that the encoded video can be accurately decoded, video encoding and decoding are performed according to the corresponding video coding standards. These video coding standards include International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-TH.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC) and other extended versions. HEVC includes Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), 3D HEVC (3D-HEVC) and other extended versions. The joint video experts team (JVET) of ITU-T and ISO / IEC has begun to develop a video coding standard called Versatile Video Coding (VVC). VVC is included in the Working Draft (WD), which includes JVET-M1001-v6, providing algorithm descriptions, encoding end descriptions of VVC WD, and reference software.

[0045] A video decoding system can encode video by using IRAP pictures and non-IRAP pictures. IRAP pictures are pictures that are decoded according to intra-frame prediction and are used as random access points for a video sequence. In intra-frame prediction, blocks in a picture are decoded by referring to other blocks in the same picture. This is in stark contrast to non-IRAP pictures that use inter-frame prediction. In inter-frame prediction, blocks in the current picture are decoded by referring to other blocks in a reference picture that is different from the current picture. Since IRAP pictures are decoded without reference to other pictures, IRAP pictures can be decoded without first decoding any other pictures. Therefore, the decoder can start decoding a video sequence at any IRAP picture. In contrast, non-IRAP pictures are decoded with reference to other pictures, so the decoder is generally unable to start decoding a video sequence at a non-IRAP picture. IRAP pictures also refresh the DPB. This is because IRAP pictures are the starting point of a CVS, and pictures in a CVS do not refer to pictures in previous CVSs. Therefore, IRAP pictures can also stop decoding errors related to inter-frame prediction because such errors cannot propagate through IRAP pictures. However, from a data size perspective, IRAP pictures are significantly larger than non-IRAP pictures. Therefore, a video sequence usually includes many non-IRAP pictures, with fewer IRAP pictures interspersed therein to balance decoding efficiency and functionality. For example, a 60-frame CVS may include one IRAP picture and 59 non-IRAP pictures.

[0046] In some cases, the video decoding system can be used to decode virtual reality (VR) videos, which can also be called 360-degree videos. VR videos can include a sphere of video content that is displayed as if the user is at the center of the sphere. Only a portion of the sphere (called the perspective) is displayed to the user. For example, the user can use a head mounted display (HMD) that selects and displays the perspective of the sphere based on the user's head movement. This creates the effect of being in the virtual space depicted by the video. To achieve this result, each image in the video sequence includes the entire sphere of video data for the corresponding moment. However, only a small portion of the image (e.g., a single perspective) is displayed to the user. The rest of the image will be discarded and not presented. The entire image is usually transmitted so that different perspectives can be dynamically selected and displayed based on the user's head movement. This approach may result in very large video file sizes.

[0047] To improve decoding efficiency, some systems divide images into sub-images. A sub-image is a defined spatial region of an image. Each sub-image includes the corresponding view of the image. Videos can be encoded at two or more resolutions. Each resolution is encoded into a different sub-stream. When a user streams a VR video, the decoding system can merge the sub-streams into a stream for transmission based on the current view used by the user. Specifically, the current view is obtained from the high-resolution sub-stream, and the unviewed view is obtained from the low-resolution stream. In this way, the highest quality video is displayed to the user and the low-quality video is discarded. If the user selects a new view, a lower resolution video will be displayed to the user. The decoder can request a new view to receive a higher resolution video. The encoder can then change the merging process accordingly. Once the IRAP image is reached, the decoder can start decoding the high-resolution video sequence at the new view. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0048] One concern with the above approach is that the length of time required to change resolution is based on the length of time before an IRAP picture is reached. This is because the decoder cannot start decoding a different video sequence at a non-IRAP picture, as described above. One way to reduce this delay is to include more IRAP pictures. However, this also increases the file size. In order to balance functionality with decoding efficiency, different views / sub-images can include IRAP images at different frequencies. For example, views that are more likely to be viewed may have more IRAP images than other views. For example, in a basketball context, views associated with the basket and / or midcourt may include IRAP images at a higher frequency than views viewing the stands or ceiling because the user is less likely to view such views.

[0049] This approach creates other problems. Specifically, the sub-images containing the view are part of a single image. When different sub-images have IRAP images with different frequencies, some images include both IRAP sub-images and non-IRAP sub-images. This is a problem because images are stored in the bitstream using NAL units. A NAL unit is a storage unit that includes a parameter set or slice of an image and a corresponding slice header. An access unit is a unit that includes an entire image. Therefore, an access unit includes all NAL units associated with an image. The NAL unit also includes a type that indicates the image type including the slice. In some video systems, all NAL units associated with a single image (for example, included in the same access unit) need to have the same type. Therefore, when an image includes both IRAP sub-images and non-IRAP sub-images, the NAL unit storage mechanism may stop operating correctly.

[0050] Disclosed herein is a mechanism for adjusting the NAL storage scheme to support images that include both IRAP sub-images and non-IRAP sub-images. This in turn allows VR videos that include different IRAP sub-image frequencies from different perspectives. In a first example, disclosed herein is a flag indicating whether an image is a mixed image. For example, the flag may indicate that an image includes both IRAP sub-images and non-IRAP sub-images. Based on this flag, the decoder may handle different sub-images in different ways when decoding in order to correctly decode and display the image / sub-image. The flag may be stored in a picture parameter set (PPS) and may be referred to as mixed_nalu_types_in_pic_flag.

[0051] In a second example, a flag indicating whether a picture is a mixed picture is disclosed herein. For example, the flag may indicate that the picture includes both IRAP sub-pictures and non-IRAP sub-pictures. In addition, the flag constrains the picture so that the mixed picture includes exactly two NAL unit types, including one IRAP type and one non-IRAP type. For example, the picture may include an IRAPNAL unit, the IRAP NAL unit including an Instantaneous Decoding Refresh (IDR) with a decodable random access leading picture (IDR_W_RADL), an IDR without a leading picture (IDR_N_LP), or a clean random access (CRA) NAL unit type (CRA_NUT). In addition, the picture may include a non-IRAP NAL unit, the non-IRAP NAL unit including and only including one of the following: a trailing picture NAL unit type (TRAIL_NUT), a decodable random access leading picture NAL unit type (RADL_NUT), or a skipped random access leading picture (random access skipped leading, RASL) NAL unit type (RASL_NUT). Based on this flag, the decoder can handle different sub-pictures in different ways when decoding in order to correctly decode and display the picture / sub-picture. The flag may be stored in the PPS and may be called mixed_nalu_types_in_pic_flag.

[0052] Figure 1Flowchart of an exemplary method 100 of operation for decoding a video signal. Specifically, the video signal is encoded at the encoder side. The encoding process compresses the video signal using various mechanisms to reduce the size of the video file. The smaller file size facilitates the transmission of the compressed video file to the user while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process is generally the same as the encoding process, which facilitates the decoder to reconstruct the video signal in the same manner.

[0053] In step 101, a video signal is input into an encoder. For example, the video signal can be an uncompressed video file stored in a memory. For another example, the video file can be captured by a video capture device (e.g., a camera) and encoded to support real-time streaming of the video. The video file can include an audio component and a video component at the same time. The video component includes a series of image frames that, when viewed in sequence, produce a visual effect of motion. These frames include pixels represented by light (referred to herein as brightness components (or brightness samples)) and colors (referred to as chrominance components (or color samples)). In some examples, the frames may also include depth values ​​to support three-dimensional viewing.

[0054] In step 103, the video is segmented into blocks. Segmentation includes subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), the frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels × 64 pixels). The CTU includes luma samples and chroma samples. The CTU can be divided into blocks using a coding tree, and then the blocks are recursively subdivided until a configuration structure that supports further encoding is obtained. For example, the luma component of the frame can be subdivided until each block includes a relatively uniform luminance value. In addition, the chroma component of the frame can be subdivided until each block includes a relatively uniform color value. Therefore, the content of the video frame is different, and the segmentation mechanism is different.

[0055] In step 105, various compression mechanisms are used to compress the image blocks segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction may be used. Inter-frame prediction is intended to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Therefore, there is no need to repeatedly describe blocks depicting objects in a reference frame in adjacent frames. An object, such as a table, may remain in a constant position in multiple frames. Therefore, the table is only described once, and adjacent frames can refer back to the reference frame. Pattern matching mechanisms may be used to match objects across multiple frames. In addition, moving objects may be represented across multiple frames due to, for example, object movement or camera movement. In a specific example, a video may show a car moving on the screen across multiple frames. Motion vectors may be used to describe such movement. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in one frame to the coordinates of the object in a reference frame. Therefore, inter-frame prediction may encode image blocks in a current frame as a set of motion vectors, representing the offsets between image blocks in the current frame and corresponding blocks in a reference frame.

[0056] Intra prediction encodes blocks in a common frame. Intra prediction exploits the fact that luminance and chrominance components tend to be clustered in one frame. For example, a patch of green in part of a tree tends to be adjacent to several similar patches of green. Intra prediction uses a variety of directional prediction modes (e.g., 33 modes in HEVC), plane mode, and direct current (DC) mode. Directional mode indicates that the samples of the current block are similar / identical to the samples of the neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks on a row / column (e.g., a plane) can be interpolated based on the neighboring blocks at the edge of the row. In fact, the plane mode represents the smooth transition of luminance / color between rows / columns by using a relatively constant slope in the changing value. The DC mode is used for boundary smoothing, indicating that the average values ​​of the samples of all neighboring blocks related to the angular direction of the block and the directional prediction mode are similar / identical. Therefore, the intra prediction block can represent the image block as various relationship prediction mode values ​​instead of the actual value. In addition, the inter prediction block can represent the image block as a motion vector value instead of the actual value. In both cases, the prediction block may not fully represent the image block in some cases. Any differences are stored in the residual block. The residual block can be transformed to further compress the file.

[0057] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction discussed above can create a blocky image in the decoder. In addition, a block-based prediction scheme can encode the block and then reconstruct the encoded block for later use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to the block / frame. These filters reduce these block artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce the reconstruction of the reference block artifacts, making it less likely that the artifacts will produce other artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0058] In step 109, once the video signal is segmented, compressed and filtered, the resulting data is encoded into a bitstream. The bitstream includes the above data and any indicative data that is expected to support appropriate reconstruction of the video signal in the decoder. For example, these data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in a memory and sent to a decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. The creation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107 and 109 can occur continuously and / or simultaneously over multiple frames and blocks. Figure 1 The order shown is presented for clarity and ease of discussion and is not intended to limit the video coding process to a particular order.

[0059] In step 111, the decoder receives the code stream and starts the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the code stream into corresponding syntax and video data. In step 111, the decoder uses the syntax data in the code stream to determine the segmentation of the frame. The segmentation should match the result of the block segmentation in step 103. Now describe the entropy coding / decoding used in step 111. The encoder makes many choices during the compression process, such as selecting a block segmentation scheme from multiple possible options based on the spatial positioning of the values ​​in the input image. A large number of binary bits can be used to indicate the exact option. The binary bits used in this article are binary values ​​that are treated as variables (for example, bit values ​​that can vary according to context). Entropy coding helps the encoder discard any options that are obviously not suitable for a particular situation, leaving a set of usable options. Then, a code word is assigned to each usable option. The length of the code word is based on the number of allowable options (for example, one binary bit is used for two options, and two binary bits are used for three to four options). Then, the encoder encodes the code word of the selected option. This scheme reduces the size of the codeword because the codeword size is as large as desired to uniquely indicate one option in a small subset of available options, rather than uniquely indicating an option in a possibly large set of all possible options. The decoder then decodes the option by determining the set of available options in a similar manner to the encoder. By determining the set of available options, the decoder can read the codeword and determine the selection made by the encoder.

[0060] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding prediction block to reconstruct the image block according to the segmentation. The prediction block may include an intra-frame prediction block and an inter-frame prediction block generated by the encoder in step 105. The reconstructed image block is then placed in a frame of the reconstructed video signal according to the segmentation data determined in step 111. The syntax of step 113 can also be indicated in the bitstream by entropy coding discussed above.

[0061] In step 115, filtering is performed on the frame of the reconstructed video signal in a manner similar to that of the encoder in step 107. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter may be applied to the frame to remove blocking artifacts. Once the frame is filtered, the video signal may be output to a display in step 117 for viewing by an end user.

[0062] Figure 21 is a schematic diagram of an exemplary encoding and decoding (codec) system 200 for video decoding. Specifically, the codec system 200 is capable of implementing the operating method 100. Broadly, the codec system 200 is used to describe components used in an encoder and a decoder. As discussed with respect to steps 101 and 103 in the operating method 100, the codec system 200 receives a video signal and segments the video signal to produce segmented video signals 201. Then, when acting as an encoder, the codec system 200 compresses the segmented video signals 201 into encoded bitstreams, as discussed with respect to steps 105, 107, and 109 in the method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream, as described in conjunction with steps 111, 113, 115, and 117 in the operating method 100. The codec system 200 includes an overall decoder control component 211, a transform scaling quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. Figure 2 In the figure, the black lines represent the movement of the data to be encoded / decoded, and the dotted lines represent the movement of the control data that controls the operation of other components. The components in the codec system 200 can all be present in the encoder. The decoder can include a subset of the components in the codec system 200. For example, the decoder can include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are now described.

[0063] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide the pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. The blocks can be referred to as nodes on the coding tree. A larger parent node is divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks may be included in a coding unit (CU). For example, a CU may be a sub-part of a CTU, including a luminance block, a red difference chrominance (Cr) block, and a blue difference chrominance (Cb) block and corresponding syntax instructions for the CU. The partitioning mode may include a binary tree (BT), a triple tree (TT), and a quad tree (QT) for dividing a node into two, three, or four child nodes of different shapes, respectively, depending on the partitioning mode used. The segmented video signal 201 is forwarded to the overall decoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.

[0064] The overall decoder control component 211 is used to make decisions related to encoding the images of the video sequence into the bitstream according to the application constraints. For example, the overall decoder control component 211 manages the optimization of the bitrate / bitstream size relative to the reconstruction quality. These decisions can be made based on the storage space / bandwidth availability and the image resolution request. The overall decoder control component 211 also manages the utilization of the buffer according to the transmission speed to alleviate the buffer under-load and over-load problems. In order to manage these problems, the overall decoder control component 211 manages the segmentation, prediction and filtering performed by other components. For example, the overall decoder control component 211 can dynamically increase the compression complexity to increase the resolution and bandwidth utilization, or reduce the compression complexity to reduce the resolution and bandwidth utilization. Therefore, the overall decoder control component 211 controls other components of the codec system 200 to balance the video signal reconstruction quality and bitrate issues. The overall decoder control component 211 creates control data to control the operation of other components. The control data is also forwarded to the header format and CABAC component 231 to be encoded in the bitstream, thereby indicating the parameters decoded in the decoder.

[0065] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices of the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks according to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 can perform multiple decoding processes to select an appropriate decoding mode for each video data block, etc.

[0066] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but are described separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, a motion vector may indicate the displacement of a coding object relative to a prediction block. A prediction block is a block that is found to closely match the block to be coded in terms of pixel difference. A prediction block may also be referred to as a reference block. This pixel difference may be determined by the sum of absolute difference (SAD), the sum of square difference (SSD), or other difference metrics. HEVC uses several coding objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU may be divided into multiple CTBs, which may then be divided into multiple CBs and included in a CU. A CU may be encoded as a prediction unit (PU) including prediction data and / or a transform unit (TU) including transform residual data of a CU. The motion estimation component 221 generates motion vectors, PUs, and TUs using rate-distortion analysis as part of a rate-distortion optimization process. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and may select a reference block, motion vector, etc. having an optimal rate-distortion characteristic. The optimal rate-distortion characteristic balances the quality of video reconstruction (e.g., the amount of data lost due to compression) and decoding efficiency (e.g., the size of the final encoding).

[0067] In some examples, the codec system 200 may calculate values ​​for sub-integer pixel positions of a reference image stored in the decoded image buffer component 223. For example, the video codec system 200 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Thus, the motion estimation component 221 may perform motion searches with respect to integer pixel positions and fractional pixel positions, and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates the motion vector of a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a prediction block of the reference image. The motion estimation component 221 outputs the calculated motion vector as motion data to the header format and CABAC component 231 for encoding, and outputs the motion to the motion compensation component 219.

[0068] The motion compensation performed by the motion compensation component 219 may involve obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. Likewise, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. After receiving the motion vector of the PU of the current video block, the motion compensation component 219 may locate the prediction block to which the motion vector points. Then, by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded, a pixel difference value is generated, thereby forming a residual video block. Typically, the motion estimation component 221 performs motion estimation on the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for the chroma component and the luma component. The prediction block and the residual block are forwarded to the transform scaling and quantization component 213.

[0069] The segmented video signal 201 is also sent to the intra-frame estimation component 215 and the intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but are described separately for conceptual purposes. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction on the current block based on the block in the current frame, replacing the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines the intra-frame prediction mode for encoding the current block. In some examples, the intra-frame estimation component 215 selects an appropriate intra-frame prediction mode from a plurality of tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then forwarded to the header format and CABAC component 231 for encoding.

[0070] For example, the intra-frame estimation component 215 uses rate-distortion analysis of various tested intra-frame prediction modes to calculate rate-distortion values ​​and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block encoded to generate the coded block and the code rate (e.g., number of bits) used to generate the coded block. The intra-frame estimation component 215 determines which intra-frame prediction mode obtains the best rate-distortion value for the block based on the distortion and rate calculation ratios of the various coded blocks. In addition, the intra-frame estimation component 215 can be used to encode the depth block of the depth map using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0071] When implemented on an encoder, the intra prediction component 217 may generate a residual block from the prediction block according to the selected intra prediction mode determined by the intra estimation component 215, or read the residual block from the bitstream when implemented on a decoder. The residual block includes the difference in values ​​between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra estimation component 215 and the intra prediction component 217 may perform operations on the luma component and the chroma component.

[0072] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 applies a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform to the residual block to generate a video block including residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform may transform the residual information from a pixel value domain to a transform domain, such as a frequency domain. The transform scaling and quantization component 213 is also used to scale the transformed residual information according to frequency, etc. This scaling involves applying a scaling factor to the residual information so as to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 may then scan the matrix including the quantized transform coefficients. The quantized transform coefficients are forwarded to the header format and CABAC component 231 for encoding into the codestream.

[0073] The scaling and inverse transform component 229 performs the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 performs inverse scaling, inverse transform and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, for subsequent use as a reference block, which may become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 may calculate a reference block by adding the residual block to the corresponding prediction block for motion estimation of subsequent blocks / frames. A filter is applied to the reconstructed reference block to reduce artifacts generated during the scaling, quantization and transform processes. These artifacts may produce inaccurate predictions (and generate other artifacts) when predicting subsequent blocks.

[0074] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block in the scaling and inverse transform component 229 can be combined with the corresponding prediction block in the intra-frame prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. Then, the filter can be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block. Figure 2 Like other components in the, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and can be implemented together, but are described separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to specific spatial regions and include multiple parameters to adjust how these filters are applied. The filter control analysis component 227 analyzes the reconstructed reference block to determine where these filters should be applied and sets the corresponding parameters. These data are forwarded to the header format and CABAC component 231 for encoding as filter control data. The in-loop filter component 225 applies these filters according to the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. These filters can be applied in the spatial / pixel domain (e.g., on the reconstructed pixel block) or in the frequency domain according to the example.

[0075] When operating as an encoder, the filtered reconstructed image blocks, residual blocks and / or prediction blocks are stored in the decoded image buffer component 223 for later motion estimation as described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed blocks and the filtered blocks and forwards the reconstructed blocks and the filtered blocks to the display as part of the output video signal. The decoded image buffer component 223 can be any memory device capable of storing prediction blocks, residual blocks and / or reconstructed image blocks.

[0076] The header format and CABAC component 231 receives data from various components of the codec system 200 and encodes these data into the encoded bitstream to be sent to the decoder. Specifically, the header format and CABAC component 231 generates various headers to encode control data such as overall control data and filter control data. In addition, prediction data including intra-frame prediction and motion data and residual data in the form of quantized transform coefficient data are encoded into the bitstream. The final bitstream includes all the information that the decoder wants to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also known as a codeword mapping table), a definition of the coding context of various blocks, an indication of the most likely intra-frame prediction mode, an indication of segmentation information, etc. This data can be encoded by entropy coding technology. For example, the information can be encoded by using context adaptive variable length coding (context adaptive variable length coding, CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (syntax-based context-adaptive binary arithmetic coding, SBAC), probability interval partitioning entropy (probability interval partitioning entropy, PIPE) coding or other entropy coding techniques. After entropy encoding, the encoded bitstream can be sent to another device (eg, a video decoder) or archived for later transmission or retrieval.

[0077] Figure 3 is a block diagram of an exemplary video encoder 300. The video encoder 300 may be used to implement the encoding function of the codec system 200 and / or to implement steps 101, 103, 105, 107, and / or 109 of the operating method 100. The encoder 300 segments the input video signal to generate segmented video signals 301 that are substantially similar to the segmented video signal 201. Then, the segmented video signals 301 are compressed and encoded into a bitstream by components of the encoder 300.

[0078] Specifically, the segmented video signal 301 is forwarded to the intra prediction component 317 for intra prediction. The intra prediction component 317 may be substantially similar to the intra estimation component 215 and the intra prediction component 217. The segmented video signal 301 is also forwarded to the motion compensation component 321 for inter prediction based on the reference block in the decoded image buffer 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks in the intra prediction component 317 and the motion compensation component 321 are forwarded to the transform and quantization component 313 for transforming and quantizing the residual blocks. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual blocks and the corresponding prediction blocks (and related control data) are forwarded to the entropy coding component 331 for encoding into the bitstream. The entropy coding component 331 may be substantially similar to the header format and CABAC component 231.

[0079] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 to be reconstructed as a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. According to an example, the in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. As discussed with respect to the in-loop filter component 225, the in-loop filter component 325 may include a plurality of filters. The filtered block is then stored in the decoded image buffer 323 for use as a reference block by the motion compensation component 321. The decoded image buffer 323 may be substantially similar to the decoded image buffer component 223.

[0080] Figure 4 is a block diagram of an exemplary video decoder 400. The video decoder 400 may be used to implement the decoding function of the codec system 200 and / or implement steps 111, 113, 115, and / or 117 of the operation method 100. For example, the decoder 400 receives a bitstream from the encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0081] The code stream is received by an entropy decoding component 433. Entropy decoding component 433 is used to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE decoding or other entropy coding techniques. For example, entropy decoding component 433 can use header information to provide context to interpret other data encoded as code words in the code stream. The decoded information includes any information required to decode the video signal, such as overall control data, filter control data, segmentation information, motion data, prediction data, and quantized transform coefficients of residual blocks. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 to be reconstructed as residual blocks. The inverse transform and quantization component 429 can be substantially similar to the inverse transform and quantization component 329.

[0082] The reconstructed residual block and / or prediction block are forwarded to the intra prediction component 417 to be reconstructed into an image block according to the intra prediction operation. The intra prediction component 417 may be similar to the intra estimation component 215 and the intra prediction component 217. Specifically, the intra prediction component 417 uses the prediction mode to locate the reference block in the frame and applies the residual block to the result to reconstruct the intra prediction image block. The reconstructed intra prediction image block and / or residual block and the corresponding inter prediction data are forwarded to the decoded image buffer component 423 through the in-loop filter component 425, and the decoded image buffer component 423 and the in-loop filter component 425 may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block and / or prediction block, and this information is stored in the decoded image buffer component 423. The reconstructed image block in the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses the motion vector in the reference block to generate a prediction block and applies the residual block to the result to reconstruct the image block. The resulting reconstructed block may also be forwarded to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 continues to store other reconstructed image blocks, which may be reconstructed into frames using the segmentation information. These frames may also be arranged in sequence. The sequence is output to the display screen as a reconstructed output video signal.

[0083] Figure 55 is a schematic diagram of an exemplary CVS 500. For example, according to method 100, CVS 500 can be encoded by an encoder such as codec system 200 and / or encoder 300. In addition, CVS 500 can be decoded by a decoder such as codec system 200 and / or decoder 400. CVS 500 includes images decoded in decoding order 508. Decoding order 508 is the order in which images are located in the code stream. Then, the images in CVS 500 are output in presentation order 510. Presentation order 510 is the order in which the decoder can display the images to cause the generated video to be correctly displayed. For example, the images in CVS 500 can generally be located in presentation order 510. However, some images can be moved to different positions to improve decoding efficiency, such as by placing similar images closer together to support inter-frame prediction. Moving these images in this way produces decoding order 508. In the example shown, the images are indexed from 0 to 4 in decoding order 508. In presentation order 510 , images at index 2 and index 3 have been moved in front of the image at index 0 .

[0084] CVS 500 includes IRAP picture 502. IRAP picture 502 is an image decoded according to intra-frame prediction and is used as a random access point of CVS 500. Specifically, blocks in IRAP picture 502 are decoded by referring to other blocks in IRAP picture 502. Since IRAP picture 502 is decoded without referring to other pictures, IRAP picture 502 can be decoded without first decoding any other pictures. Therefore, the decoder can start decoding CVS 500 at IRAP picture 502. In addition, IRAP picture 502 can cause DPB to be refreshed. For example, the picture presented after IRAP picture 502 can be inter-frame predicted without relying on the picture before IRAP picture 502 (e.g., picture index 0). Therefore, once IRAP picture 502 is decoded, the picture buffer can be refreshed. This has the effect of stopping any decoding errors related to inter-frame prediction because these errors cannot be propagated through IRAP picture 502. IRAP picture 502 can include various types of pictures. For example, IRAP picture can be decoded as IDR or CRA. IDR is an intra-frame decoded picture that starts a new CVS 500 and flushes the picture buffer. CRA is an intra-frame decoded picture that acts as a random access point without starting a new CVS 500 or flushing the picture buffer. In this way, the previous picture 504 associated with CRA can refer to the picture before CRA, while the previous picture 504 associated with IDR may not refer to the picture before IDR.

[0085] CVS 500 also includes various non-IRAP pictures. These non-IRAP pictures include leading pictures 504 and trailing pictures 506. Leading pictures 504 are pictures that follow IRAP pictures 502 in decoding order 508 but precede IRAP pictures 502 in presentation order 510. Trailing pictures 506 follow IRAP pictures 502 in decoding order 508 and presentation order 510. Both leading pictures 504 and trailing pictures 506 are decoded according to inter-frame prediction. Trailing pictures 506 are decoded with reference to IRAP pictures 502 or pictures that follow IRAP pictures 502. Therefore, once IRAP pictures 502 are decoded, trailing pictures 506 can always be decoded. Leading pictures 504 can include random access skipped leading (RASL) pictures and decodable random access decodable leading (RADL) pictures. RASL pictures are decoded by referring to pictures before IRAP picture 502, but are decoded at a position after IRAP picture 502. Since RASL pictures are dependent on pictures before, when the decoder starts decoding at IRAP picture 502, RASL pictures cannot be decoded. Therefore, when IRAP picture 502 is used as a random access point, RASL pictures are skipped and not decoded. However, when the decoder uses the previous IRAP picture (before index 0 and not displayed) as a random access point, the RASL picture is decoded and displayed. RADL pictures are decoded with reference to IRAP picture 502 and / or pictures after IRAP picture 502, but are positioned before IRAP picture 502 in presentation order 510. Since RADL pictures are not dependent on pictures before IRAP picture 502, when IRAP picture 502 is a random access point, RADL pictures can be decoded and displayed.

[0086] All images in CVS 500 can be stored in access units. In addition, images can be divided into slices, and the slices can be included in NAL units. NAL units are storage units that include parameter sets or slices of images and corresponding slice headers. NAL units are assigned types to indicate to the decoder the type of data included in the NAL unit. For example, the slices in the IRAP image 502 can be included in an IDR with RADL (IDR_W_RADL) NAL unit, an IDR without a preceding image (IDR_N_LP) NAL unit, a CRA NAL unit, etc. The IDR_W_RADL NAL unit indicates that the IRAP image 502 is an IDR image associated with a RADL preceding image 504. The IDR_N_LP NAL unit indicates that the IRAP image 502 is an IDR image that is not associated with any preceding image 504. The CRANAL unit indicates that the IRAP image 502 is a CRA image that can be associated with a preceding image 504. The slices in non-IRAP images can also be placed in NAL units. For example, a slice in the trailing picture 506 may be placed in a trailing picture NAL unit type (TRAIL_NUT) to indicate that the trailing picture 506 is an inter-frame prediction decoded picture. A slice in the leading picture 504 may be included in a RASL NAL unit type (RASL_NUT) and / or a RADL NAL unit type (RADL_NUT) to indicate that the corresponding picture is the corresponding type of inter-frame prediction decoded leading picture 504. By signaling the slices in the picture in the corresponding NAL unit, the decoder can easily determine the appropriate decoding mechanism to apply to each picture / slice.

[0087] Figure 6 Schematic diagram of a plurality of sub-image video streams 601, 602, and 603 divided from a VR image video stream 600. For example, each of the sub-image video streams 601 to 603 and / or the VR image video stream 600 can be decoded into the CVS 500. Therefore, according to the method 100, the sub-image video streams 601 to 603 and / or the VR image video stream 600 can be encoded by an encoder such as the codec system 200 and / or the encoder 300. In addition, the sub-image video streams 601 to 603 and / or the VR image video stream 600 can be decoded by a decoder such as the codec system 200 and / or the decoder 400.

[0088] The VR image video stream 600 includes multiple images presented over time. Specifically, VR operates by decoding a sphere of video content, which is displayed as if the user is at the center of the sphere. Each image includes the entire sphere. At the same time, only a portion of the image (called the perspective) is displayed to the user. For example, the user can use a head mounted display (HMD) that selects and displays the perspective of the sphere based on the user's head movement. This creates the effect of being in the virtual space depicted by the video. To achieve this result, each image in the video sequence includes the entire sphere of video data for the corresponding moment. However, only a small portion of the image (e.g., a single perspective) is displayed to the user. The rest of the image will be discarded and not presented. The entire image is usually transmitted so that different perspectives can be dynamically selected and displayed based on the user's head movement.

[0089] In the example shown, the images of the VR image video stream 600 can be subdivided into sub-images according to the available viewing angle. Therefore, each image and the corresponding sub-image include a time position (e.g., image order) as part of the time representation. The sub-image video streams 601 to 603 are created when the subdivision is performed consistently over time. This consistent subdivision produces sub-image video streams 601 to 603, each of which includes a set of sub-images having a predetermined size, shape, and spatial position relative to the corresponding image in the VR image video stream 600. In addition, the time position of the sub-image set in the sub-image video streams 601 to 603 changes during the presentation time. In this way, the sub-images in the sub-image video streams 601 to 603 can be aligned in the time domain according to the time position. Then, the sub-images of the sub-image video signals 601 to 603 from each time position can be merged in the spatial domain according to the predefined spatial position to reconstruct the VR image video stream 600 for display. Specifically, the sub-image video streams 601 to 603 can be encoded into separate sub-code streams, respectively. When these sub-streams are merged together, they produce a single stream that includes the entire set of images over time. The resulting stream can be transmitted to a decoder for decoding and display based on the user's currently selected viewing angle.

[0090] One problem with VR video is that all sub-image video streams 601 to 603 can be transmitted to the user at high quality (e.g., high resolution). This allows the decoder to dynamically select the user's current perspective and display the sub-images in the corresponding sub-image video streams 601 to 603 in real time. However, the user may only view a single perspective, such as from sub-image video stream 601, while sub-image video streams 602 and 603 are discarded. Therefore, transmitting sub-image video streams 602 and 603 at high quality may waste a lot of bandwidth. To improve decoding efficiency, VR video can be encoded into multiple video streams 600, each of which is encoded at a different quality / resolution. In this way, the decoder can transmit a request for the current sub-image video stream 601. In response, the encoder (or mid-striper or other content server) can select a higher quality sub-image video stream 601 from the higher quality video stream 600 and a lower quality sub-image video stream 602 and 603 from the lower quality video stream 600. The encoder can then merge these sub-streams into a complete encoded stream for transmission to the decoder. In this way, the decoder receives a series of images, where the current view has a higher quality and the other views have a lower quality. In addition, the highest quality sub-image is usually displayed to the user (without head movement), and the low-quality sub-image is usually discarded, so that the function and decoding efficiency are balanced.

[0091] In the case where the user switches from watching the sub-image video stream 601 to the sub-image video stream 602, the decoder requests the transmission of the new current sub-image video stream 602 at a higher quality. The encoder can then change the merging mechanism accordingly. As described above, the decoder can only start decoding the new CVS 500 at the IRAP picture 502. Therefore, the sub-image video stream 602 is displayed at a lower quality until the IRAP picture / sub-image is reached. The IRAP picture can then be decoded at a higher quality to start decoding a higher quality version of the sub-image video stream 602. This approach significantly improves video compression without adversely affecting the user's viewing experience.

[0092] One concern about the above approach is that the length of time required to change the resolution is based on the length of time before an IRAP picture is reached in the video stream. This is because the decoder cannot start decoding different versions of the sub-image video stream 602 at a non-IRAP picture. One way to reduce this delay is to include more IRAP pictures. However, this also increases the file size. In order to balance functionality and decoding efficiency, different perspective / sub-image video streams 601 to 603 may include IRAP pictures of different frequencies. For example, a perspective / sub-image video stream 601 to 603 that is more likely to be viewed may have more IRAP pictures than other perspective / sub-image video streams 601 to 603. For example, in a basketball background, a perspective / sub-image video stream 601 to 603 associated with a basket and / or midfield may include more frequent IRAP pictures than a perspective / sub-image video stream 601 to 603 viewing the stands or ceiling, so that the user is less likely to view the perspective / sub-image video stream 601 to 603.

[0093] This approach can create other problems. Specifically, the sub-images in the sub-image video streams 601 to 603 that share a POC are part of a single image. As described above, the slices in the image are included in the NAL unit according to the image type. In some video decoding systems, all NAL units associated with a single image are restricted to include the same NAL unit type. When different sub-image video streams 601 to 603 have IRAP images of different frequencies, some images include both IRAP sub-images and non-IRAP sub-images. This violates the constraint that each image can only use the same type of NAL unit.

[0094] The present invention solves this problem by removing the constraint that all NAL units of the slices in the image use the same NAL unit type. For example, the image is included in the access unit. By removing this constraint, the access unit can include an IRAP NAL unit type and a non-IRAP NAL unit type. In addition, a flag can be encoded to indicate when the image / access unit includes a mixed image / access unit of an IRAP NAL unit type and a non-IRAP NAL unit type. In some examples, the flag is a mixed NAL unit type (mixed_nalu_types_in_pic_flag) in the image flag. In addition, a constraint can be applied requiring that a single mixed image / access unit can only include one type of IRAP NAL unit and one type of non-IRAP NAL unit. This can prevent accidental NAL unit type mixing from occurring. If such a mixed situation can exist, the decoder must be designed to manage such a mixed situation. This will unnecessarily increase the required hardware complexity without providing additional benefits to the encoding process. For example, a mixed image can include an IRAP NAL unit of one type selected from IDR_W_RADL, IDR_N_LP or CRA_NUT. Furthermore, the hybrid picture may include a non-IRAP NAL unit of a type selected from TRAIL_NUT, RADL_NUT and RASL_NUT.Exemplary implementations of this scheme will be discussed in more detail below.

[0095] Figure 7 is a schematic diagram of an exemplary bitstream 700 including an image with mixed NAL unit types. For example, the bitstream 700 may be generated by the codec system 200 and / or the encoder 300, and decoded by the codec system 200 and / or the decoder 400 according to the method 100. In addition, the bitstream 700 may include a VR image video stream 600 merged from a plurality of sub-image video streams 601 to 603 at a plurality of video resolutions, wherein each sub-image video stream includes a CVS 500 located at a different spatial position.

[0096] The code stream 700 includes a sequence parameter set (SPS) 710, a plurality of picture parameter sets (PPS) 711, a plurality of slice headers 715, and image data 720. The SPS 710 includes sequence data common to all images in the video sequence included in the code stream 700. Such data may include image size, bit depth, decoding tool parameters, bit rate limits, etc. The PPS 711 includes parameters applied to the entire image. Therefore, each image in the video sequence may refer to the PPS 711. It should be noted that, although each image refers to the PPS 711, in some examples, a single PPS 711 may include data for a plurality of images. For example, a plurality of similar images may be encoded according to similar parameters. In this case, a single PPS 711 may include data for such similar images. The PPS 711 may indicate decoding tools, quantization parameters, offsets, etc. that may be used for the slices in the corresponding image. The slice header 715 includes parameters specific to each slice in the image. Therefore, each slice in the video sequence may have a slice header 715. The slice header 715 may include slice type information, picture order count (POC), reference picture list, prediction weight, partition entry point, deblocking filter parameters, etc. It should be noted that in some contexts, the slice header 715 may also be referred to as a partition group header.

[0097] The image data 720 includes video data encoded according to inter-frame prediction and / or intra-frame prediction and corresponding transformed and quantized residual data. For example, a video sequence includes multiple images 721 decoded as image data 720. Image 721 is a single frame in a video sequence, and therefore is usually displayed as a single unit when the video sequence is displayed. However, a sub-image 723 can be displayed to implement technologies such as virtual reality. Multiple images 721 each refer to PPS 711. Image 721 can be divided into sub-images 723, blocks and / or strips. Sub-image 723 is a spatial region of image 721 that is consistently applied to the encoded video sequence. Therefore, sub-image 723 can be displayed by the HMD in a VR context. In addition, a sub-image 723 with a specified POC can be obtained from the sub-image video streams 601 to 603 of the corresponding resolution. The sub-image 723 can refer to SPS 710. In some systems, a strip 725 is referred to as a block group including blocks. The slice 725 and / or the partition group of the partition refer to the slice header 715. The slice 725 can be defined as an integer number of complete partitions or an integer number of continuous complete CTU rows within the partition of the image 721, which are included only in a single NAL unit. Therefore, the slice 725 is further divided into CTU and / or CTB. According to the coding tree, the CTU / CTB is further divided into coding blocks. The coding blocks can then be encoded / decoded according to the prediction mechanism.

[0098] Parameter sets and / or slices 725 are decoded in NAL units. NAL units can be defined as syntax structures including an indication of the data type to be followed, and bytes in the form of RBSPs including the data, interspersed with aliasing prevention bytes when necessary. More specifically, a NAL unit is a storage unit including parameter sets or slices 725 in an image 721 and corresponding slice headers 715. Specifically, a VCL NAL unit 740 is a NAL unit including slices 725 in an image 721 and corresponding slice headers 715. In addition, a non-VCL NAL unit 730 includes parameter sets, such as SPS 710 and PPS 711. Several types of NAL units can be used. For example, SPS 710 and PPS 711 can be included in SPS NAL unit types (SPS_NUT) 731 and PPS NAL unit types (PPS_NUT) 732, respectively, both of which are non-VCL NAL units 730.

[0099] As described above, an IRAP image such as the IRAP image 502 may be included in the IRAP NAL unit 745. Non-IRAP images such as the preceding image 504 and the succeeding image 506 may be included in the non-IRAP NAL unit 749. Specifically, the IRAP NAL unit 745 is any NAL unit including a slice 725 obtained from an IRAP image or a sub-image. The non-IRAP NAL unit 749 is any NAL unit including a slice 725 obtained from any image that is not an IRAP image or a sub-image (e.g., a preceding image or a succeeding image). Both the IRAP NAL unit 745 and the non-IRAP NAL unit 749 are VCL NAL units 740 because they both include slice data. In an exemplary embodiment, the IRAP NAL unit 745 may include a slice 725 from an IDR image without a preceding image or an IDR image associated with a RADL image in an IDR_N_LP NAL unit 741 or an IDR_w_RADL NAL unit 742, respectively. Additionally, IRAP NAL unit 745 may include slices 725 from a CRA picture in CRA_NUT 743. In an exemplary embodiment, non-IRAP NAL unit 749 may include slices 725 from a RASL picture, a RADL picture, or a trailing picture in RASL_NUT 746, RADL_NUT 747, or TRAIL_NUT 748, respectively. A complete list of possible NAL units, sorted by NAL unit type, is shown below in an exemplary embodiment.

[0100]

[0101]

[0102] As described above, the VR video stream may include sub-images 723 having IRAP images of different frequencies. In this way, spatial areas that are less likely to be viewed by the user use fewer IRAP images, while spatial areas that the user may view frequently use more IRAP images. In this way, spatial areas that the user may switch back to regularly can be quickly adjusted to a higher resolution. When this method causes the image 721 to include IRAP NAL units 745 and non-IRAP NAL units 749, the image 721 is called a mixed image. This condition can be indicated by the mixed NAL unit type (mixed_nalu_types_in_pic_flag) in the image flag 727. The mixed_nalu_types_in_pic_flag 727 can be set in the PPS 711. In addition, when each image 721 representing the reference PPS 711 has multiple VCL NAL units 740 and the VCL NAL units 740 do not have the same NAL unit type (nal_unit_type) value, the mixed_nalu_types_in_pic_flag 727 can be set to 1. Furthermore, when it is indicated that each picture 721 referring to the PPS 711 has one or more VCL NAL units 740 and the VCL NAL units 740 of each picture 721 referring to the PPS 711 have the same nal_unit_type value, the mixed_nalu_types_in_pic_flag 727 may be set to 0.

[0103] In addition, a constraint may be used that when mixed_nalu_types_in_pic_flag 727 is set, the VCL NAL units 740 of one or more sub-pictures 723 in the picture 721 all have a first specific NAL unit type value, and the other VCL NAL units 740 in the picture 721 all have a different second specific NAL unit type value. For example, the constraint may require that the mixed picture 721 include a single type of IRAP NAL unit 745 and a single type of non-IRAP NAL unit 749. For example, the picture 721 may include one or more IDR_N_LP NAL units 741, one or more IDR_w_RADL NAL units 742, or one or more CRA_NUTs 743, but not any combination of these IRAP NAL units 745. In addition, the picture 721 may include one or more RASL_NUTs 746, one or more RADL_NUTs 747, or one or more TRAIL_NUTs 748, but not any combination of these IRAP NAL units 745.

[0104] In an exemplary implementation, the picture type is used to define the decoding process. These processes include, for example, deriving the picture identification by the picture order count (POC), marking the reference picture status in the decoded picture buffer (DPB), outputting the picture from the DPB, etc. The picture can be identified by type based on the NAL unit type including all coded pictures or their sub-parts. In some video decoding systems, the picture type may include instantaneous decoding refresh (IDR) pictures and non-IDR pictures. In other video decoding systems, the picture type may include post-pictures, temporal sub-layer access (TSA) pictures, step-wise temporal sub-layer access (STSA) pictures, decodable random access leading (RADL) pictures, skipped random access leading (RASL) pictures, broken-link access (BLA) pictures, instantaneous random access pictures, and pure random access pictures. These picture types can be further differentiated based on whether the picture is a sub-layer reference picture or a sub-layer non-reference picture. BLA pictures can be further differentiated into BLA with a preceding picture, BLA with a RADL picture, and BLA without a preceding picture. IDR pictures can be further differentiated into IDR with a RADL picture and IDR without a preceding picture.

[0105] These image types can be used to implement various video-related functions. For example, IRAP images can be implemented using IDR, BLA and / or CRA images. IRAP images may provide the following functions / benefits. The presence of an IRAP image may indicate that the decoding process can be started from this image. This functionality allows the implementation of a random access function in which the decoding process starts at a specified position in the codestream as long as the IRAP image exists at that position. Such a position is not necessary at the beginning of the codestream. The presence of an IRAP image also refreshes the decoding process so that the coded images starting from the IRAP image (excluding RASL images) are decoded without reference to the images preceding the IRAP image. Therefore, the IRAP image located in the codestream stops propagating decoding errors. Therefore, decoding errors of coded images preceding the IRAP image cannot be propagated through the IRAP image and in the decoding order to the images following the IRAP image.

[0106] IRAP pictures provide various functions but have an adverse impact on compression efficiency. Therefore, the presence of IRAP pictures may cause a sharp increase in bit rate. There are many reasons for this adverse impact on compression efficiency. For example, an IRAP picture is an intra-frame prediction picture that uses much more bits to represent than an inter-frame prediction picture used as a non-IRAP picture. In addition, the presence of IRAP pictures violates the temporal prediction used in inter-frame prediction. Specifically, IRAP pictures refresh the decoding process by removing previous reference pictures from the DPB. Removing previous reference pictures reduces the availability of reference pictures to decode pictures that follow the IRAP picture in the decoding order, thereby reducing the efficiency of this process.

[0107] IDR images may use different signaling and derivation processes than other IRAP image types. For example, the IDR-related indication and derivation process may set the most significant bit (MSB) portion of the POC to 0 instead of deriving the MSB from the previous key image. In addition, the slice header of the IDR image may not include information to assist in reference image management. At the same time, other image types, such as CRA, post-position, TSA, etc., may include reference image information, such as a reference picture set (RPS) or a reference picture list, which can be used to implement a reference image marking process. The reference image marking process is the process of determining whether the status of a reference image in the DPB is used for reference or not used for reference. For IDR images, this information may not be signaled because the presence of an IDR indicates that the decoding process should only mark all reference images in the DPB as unused reference images.

[0108] In addition to the image type, the image identifier of the POC is also used for various purposes, such as for managing reference images in inter-frame prediction, for outputting images from the DPB, for motion vector scaling, for weighted prediction, etc. For example, in some video decoding systems, images in the DPB can be marked as used for short-term reference, for long-term reference, or for unused reference. Once an image is marked as unused for reference, the image can no longer be used for prediction. When such an image is no longer needed to be output, the image can be deleted from the DPB. In other video decoding systems, reference images can be marked as short-term and long-term references. When the image is no longer needed for prediction reference, the reference image can be marked as unused for reference. The transition between these states can be controlled by the decoded reference image marking process. An implicit sliding window process and / or an explicit memory management control operation (MMCO) process can be used as a decoding reference image marking mechanism. When the number of reference frames is equal to the given maximum number (max_num_ref_frames in SPS), the sliding window process marks the short-term reference image as unused for reference. The short-term reference pictures are stored in a first-in-first-out manner so that the most recently decoded short-term pictures are stored in the DPB. The explicit MMCO process may include multiple MMCO commands. The MMCO command may mark one or more short-term or long-term reference pictures as "not used for reference", mark all pictures as "not used for reference", or mark the current reference picture or an existing short-term reference picture as "long-term reference", and assign a long-term picture index to the long-term reference picture.

[0109] In some video coding systems, the reference picture marking operation and the process of outputting and deleting pictures from the DPB are performed after the picture is decoded. Other video coding systems use RPS for reference picture management. The most fundamental difference between the RPS mechanism and the MMCO / sliding window process is that, for each specific slice, the RPS provides a complete set of reference pictures used by the current picture or any subsequent picture. Therefore, the complete set of all pictures that must be kept in the DPB for use by the current or future pictures is indicated in the RPS. This is different from the MMCO / sliding window scheme, which only indicates relative changes to the DPB. In the RPS concept, it is not necessary to obtain information from earlier pictures in the decoding order to maintain the correct state of the reference pictures in the DPB. In some video coding systems, in order to take advantage of RPS and improve error resilience, the order of picture decoding and DPB operations is changed. In some video coding systems, picture marking and buffering operations, outputting and deleting decoded pictures from the DPB are usually applied after the current picture is decoded. In other video coding systems, the RPS first decodes from the slice header of the current picture, and then typically performs picture marking and buffering operations before decoding the current picture.

[0110] In VVC, the reference image management method can be summarized as follows. Two reference image lists are directly indicated and derived, denoted as List 0 and List 1. The two reference lists are not based on the RPS or sliding window plus MMCO process as described above. Reference image marking is directly based on reference image lists 0 and 1, using active and inactive entries in the reference image lists, while in inter-frame prediction of CTU, only active entries can be used as reference indexes. The information used to derive the two reference image lists is signaled by the syntax elements and syntax structures in the SPS, PPS and slice header. The predefined RPL structure is indicated in the SPS for use by reference in the slice header. Two reference image lists are generated for all types of slices, including bidirectional inter-frame prediction (B) slices, unidirectional inter-frame prediction (P) slices, and intra-frame prediction (I) slices. The two reference image lists can be constructed without using the reference image list initialization process or the reference image list modification process. The long-term reference picture (LTRP) is identified by the POC LSB. The incremental POC MSB period may be an indication of the LTRP, as determined on a picture-by-picture basis.

[0111] In order to encode a video image, the image is first segmented and each segment is encoded into a bitstream. There are many image segmentation schemes. For example, an image can be segmented into regular strips, non-independent strips, tiles, and / or segmented according to wavefront parallel processing (WPP). For simplicity, HEVC restricts the encoder so that only regular strips, non-independent strips, tiles, WPP, and combinations thereof can be used when segmenting strips into CTB groups for video coding. This segmentation method can be used to support maximum transfer unit (MTU) size matching, parallel processing, and reduce end-to-end delay. MTU represents the maximum amount of data that can be transmitted by a single data packet. If the data packet payload exceeds the MTU, the data packet payload is divided into two data packets through a fragmentation process.

[0112] A regular slice, also referred to as a slice, is a portion of an image obtained after segmentation, which can be reconstructed independently of other regular slices in the same image, but there is also a mutual dependency due to the existence of loop filtering operations. Each regular slice is encapsulated in its own Network Abstraction Layer (NAL) unit for transmission. In addition, in order to support independent reconstruction, intra-frame prediction (intra-frame sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries can be disabled. This independent reconstruction supports parallelization. For example, parallelization based on regular slices reduces inter-processor or inter-core communication. However, since regular slices are independent, each slice is associated with a separate slice header. Since each slice has a slice header bit overhead and lacks prediction across slice boundaries, the use of regular slices generates a large amount of coding overhead. In addition, regular slices can be used to support matching MTU size requirements. Specifically, since regular slices are encapsulated in separate NAL units and can be independently encoded, each regular slice needs to be smaller than the MTU in the MTU scheme to avoid the slice being split into multiple data packets. Therefore, in order to achieve parallelization MTU size matching, the stripe layouts in the image will contradict each other.

[0113] A non-independent slice is similar to a regular slice, but the slice header is shortened, and the image tree block boundary can be segmented without destroying inter-frame prediction. Accordingly, a non-independent slice allows a regular slice to be dispersed into multiple NAL units, so that a part of the regular slice can be sent out before the encoding of the entire regular slice is completed, thereby reducing end-to-end delay.

[0114] An image can be divided into block groups / strips and blocks. A block is a sequence of CTUs covering a rectangular area of ​​an image. A block group / strip includes multiple blocks of an image. Raster scan block group mode and rectangular block group mode can be used to create blocks. In raster scan block group mode, a block group includes a sequence of blocks in a block raster scan of an image. In rectangular strip block group mode, a block group includes several blocks of an image, which together form a rectangular area of ​​the image. The blocks in a rectangular block group are arranged in the block raster scan order of the block group. For example, a block can be a partition of an image formed by horizontal and vertical boundaries, which form columns and rows of the block. The blocks can be encoded in raster scan order (from right to left, from top to bottom). The scan order of CTBs is the order in which the scan is performed within a block. Accordingly, the CTBs in the first block are encoded in raster scan order before proceeding to the CTBs in the next block. Similar to conventional slices, partitioning eliminates the reliance on inter-frame prediction and entropy decoding. However, partitions may not be included in individual NAL units, and therefore, partitions cannot be used for MTU size matching. Each partition can be processed by one processor / core, and the inter-processor / inter-core communication used for inter-frame prediction between processing units that decode adjacent partitions can be limited to transmitting shared slice headers (when adjacent partitions are in the same slice) and sharing reconstructed samples and metadata related to loop filtering. When a slice includes multiple partitions, the entry point byte offset of each partition except the first entry point offset in the slice can be indicated in the slice header. For each slice and partition, at least one of the following conditions needs to be met: (1) all coding tree blocks in the slice belong to the same partition; (2) all coding tree blocks in the partition belong to the same slice.

[0115] In WPP, the image is partitioned into a single row of CTBs. Entropy decoding and prediction mechanisms can use data from CTBs in other rows. Parallel processing is achieved by parallel decoding of CTB rows. For example, the current row can be decoded in parallel with the previous row. However, the decoding of the current row is delayed by two CTBs compared to the decoding process of the previous rows. This delay ensures that data related to the CTBs above and to the right of the current CTB in the current row are available before the current CTB is decoded. When represented graphically, this approach is shown as a wavefront. This staggered start decoding can be parallelized using as many processors / cores as the CTB rows included in the image. Since inter-frame prediction is allowed between adjacent tree block rows within the image, a large amount of inter-processor / inter-core communication may be required to achieve intra-frame prediction. WPP partitioning does not take into account the NAL unit size. Therefore, WPP does not support MTU size matching. However, conventional strips can be used in conjunction with WPP with certain decoding overhead to achieve MTU size matching as needed. Finally, the wavefront segment can contain exactly one CTB row. Furthermore, when using WPP, when a stripe starts in a CTB row, the stripe should end in the same CTB row.

[0116] Tiles may also include motion constrained tilesets. A motion constrained tileset (MCTS) is a tileset designed to restrict the associated motion vectors to point to whole sample positions inside the MCTS and to fractional sample positions that only require interpolation of whole sample positions inside the MCTS. In addition, motion vector candidates derived from blocks outside the MCTS are not allowed to be used for temporal motion vector prediction. In this way, each MCTS can be decoded independently when all tiles are included in the MCTS. A temporal MCTS supplemental enhancement information (SEI) message can be used to indicate the presence of an MCTS in a bitstream and to indicate the MCTS. The MCTS SEI message provides supplemental information (detailed as part of the SEI message semantics) that can be used to extract the MCTS sub-codestream to generate a consistent codestream for the MCTS set. The information includes multiple extraction information sets, each of which defines multiple MCTS sets and includes raw bytes sequence payload (RBSP) bytes of replacement video parameter sets (VPS), sequence parameter sets (SPS) and picture parameter sets (PPS) to be used in the MCTS substream extraction process. Because one or all of the slice address-related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) can take different values ​​in the extracted substream, the parameter sets (VPS, SPS, PPS) can be rewritten or replaced when extracting the substream according to the MCTS substream extraction process, and the slice header is updated.

[0117] VR applications, also known as 360-degree video applications, can display only a portion of a complete sphere, and therefore only a subset of the entire image. Perspective-dependent 360 transmission of dynamic adaptive streaming based on the hypertext transfer protocol (DASH) mechanism can be used to reduce the bit rate and support the transmission of 360-degree video through a streaming mechanism. This mechanism divides the sphere / projection image into multiple MCTS, for example by using a cubemap projection (CMP). Two or more streams can be encoded with different spatial resolutions or qualities. When the data is transmitted to the decoder, the MCTS from the high-resolution / quality stream is transmitted for the perspective to be displayed (e.g., the front perspective). MCTS from the low-resolution / quality stream is transmitted for other perspectives. These MCTS are packaged in a certain way and then sent to the receiver for decoding. The perspective that the user is expected to view is represented by a high-resolution / high-quality MCTS to create a frontal viewing experience. When the user's head turns to view another perspective (e.g., left or right perspective), the displayed content is from a lower resolution / quality perspective for a short period of time when the system is acquiring a high-resolution / quality MCTS for the new perspective. When the user's head rotates to see another view, there is a delay between the time the user's head rotates and the time they see a higher resolution / quality representation of the view. This delay depends on how fast the system can fetch a higher resolution / quality MCTS for that view, which in turn depends on the IRAP cycle. The IRAP cycle is the interval between two IRAP occurrences. This delay is related to the IRAP cycle because the MCTS for the new view can only be decoded starting from the IRAP image.

[0118] For example, if the IRAP cycle is encoded once every second, the following applies. If the user's head turns to see the new view before the system starts fetching new segments / IRAP cycles, the best case for latency is the same as the network round trip latency. In this case, the system will be able to request a higher resolution / quality MCTS for the new view immediately, so the only latency is the network round trip latency, which is the latency of the fetch request plus the transmission time of the requested MCTS, assuming that the minimum buffering delay can be set to about 0 and the sensor latency is small and can be ignored. For example, the network round trip latency can be around 200 milliseconds. If the user's head turns to see the new view after the system has just requested the next segment, the worst case for latency is the IRAP cycle + the network round trip latency. The codestream can be encoded with more frequent IRAP images, making the IRAP cycle shorter, to improve the worst case above, as this reduces the overall latency. However, this approach increases bandwidth requirements as compression efficiency decreases.

[0119] In an exemplary implementation, sub-images in the same decoded image are allowed to include different nal_unit_type values. This mechanism is described as follows. An image can be divided into sub-images. A sub-image is a set of rectangular tile groups / strips, starting with a tile group whose tile_group_address is equal to 0. Each sub-image can refer to a corresponding PPS and therefore can have a separate tile partition. The presence of a sub-image can be indicated in the PPS. During decoding, each sub-image is treated as an image. In-loop filtering across sub-image boundaries may always be disabled. The sub-image width and height can be expressed in units of luma CTU size. The position of the sub-image in the image may not be indicated, but can be derived using the following rules. The sub-image takes the next such unoccupied position in the image in CTU raster scan order that is large enough to contain the sub-image within the image boundary. The reference image used to decode each sub-image is generated by extracting a region collocated with the current sub-image from the reference image in the decoded image buffer. The extracted region is the decoded sub-image, so inter-frame prediction occurs between sub-images of the same size and position within the image. In this case, allowing different nal_unit_type values ​​in a coded picture enables sub-pictures originating from random access pictures and sub-pictures originating from non-random access pictures to be merged into the same coded picture without substantial difficulty (e.g., without VCL level modifications). This benefit also applies to MCTS-based coding.

[0120] In other cases, it may be beneficial to allow different nal_unit_type values ​​to be used in the coded pictures. For example, some areas of the 360 ​​video content may be viewed more frequently by a user than other areas. In order to establish a better balance between decoding efficiency and average comparable quality view switching delay in MCTS / sub-image-based view-dependent 360 video transmission, more frequent IRAP pictures may be encoded for areas that are more common than other areas. The comparable quality view switching delay is the delay experienced by a user when switching from a first view to a second view, until the presentation quality of the second view reaches a presentation quality comparable to that of the first view.

[0121] Another implementation adopts the following solution to support mixed NAL unit types in images, including POC derivation and reference picture management. There is a flag (sps_mixed_til_groups_in_pic_flag) in the parameter set, which is directly or indirectly referenced by the tile group to indicate whether there may be images with mixed IRAP and non-IRAP sub-images. For NAL units including IDR tile groups, there is a flag (poc_msb_reset_flag) in the corresponding tile group header to indicate whether the POC MSB is reset in the POC derivation of the image. A variable named PicRefreshFlag associated with the image is defined. This flag indicates whether the POC derivation and DPB status should be refreshed when decoding the image. The PicRefreshFlag value is derived as follows: If the current tile group is included in the first access unit of the codestream, PicRefreshFlag is set to 1. Otherwise, if the current tile group is an IDR tile group, PicRefreshFlag is set equal to sps_mixed_tile_groups_in_pic_flag? poc_msb_reset_flag: 1. Otherwise, if the current block group is a CRA block group, the following applies. If the current access unit is the first access unit of the decoding sequence, PicRefreshFlag is set to 1. The current access unit is the first access unit of the decoding sequence when the access unit immediately follows the end-of-sequence NAL unit or the associated variable HandleCraAsFirstPicInCvsFlag is set to 1. Otherwise, PicRefreshFlag is set to 0 (for example, the current block group does not belong to the first access unit in the codestream, nor is it an IRAP block group).

[0122] When PicRefreshFlag is equal to 1, the value of POC MSB (PicOrderCntMsb) is reset to 0 during the derivation of POC for a picture. Information used for reference picture management, such as reference picture set (RPS) or reference picture list (RPL), is indicated in the partition group / slice header regardless of the corresponding NAL unit type. The reference picture list is built at the start of decoding of each partition group, regardless of the NAL unit type. The reference picture list may include RefPicList[0] and RefPicList[1] for the RPL method, RefPicList0[] and RefPicList1[] for the RPS method, or similar lists, including reference pictures used for inter prediction operations of the picture. When PicRefreshFlag is equal to 1, during the reference picture marking process, all reference pictures in the DPB are marked as "unused for reference".

[0123] This implementation is associated with certain problems. For example, when nal_unit_type values ​​cannot be mixed in an image, when the derivation of whether an image is an IRAP image and the derivation of the variable NoRaslOutputFlag are described at the image level, the decoder can perform these derivations after receiving the first VCL NAL unit of any image. However, due to the support of mixed NAL unit types in images, the decoder must wait for the arrival of other VCL NAL units of the image before performing the above derivation. In the worst case, the decoder must wait for the arrival of the last VCL NAL unit of the image. In addition, these systems can indicate a flag in the block group header of the IDR NAL unit to indicate whether the POC MSB is reset in the POC derivation of the image. The mechanism has the following problems. This mechanism does not support the case of mixing CRANAL unit types and non-IRAP NAL unit types. Furthermore, when the state of whether IRAP (IDR or CRA) NAL units are mixed with non-IRAP NAL units in a picture changes, signaling this information in the chunk group / slice header of the VCL NAL unit would require changing the value during bitstream extraction or merging. The rewriting of the slice header occurs each time a user requests video, thus requiring significant hardware resources. Furthermore, in addition to a mix of specific IRAP NAL unit types and specific non-IRAP NAL unit types, there can be some other mixes of different NAL unit types in a picture. Such flexibilities do not provide support for practical use cases, while they complicate the design of the codec, which unnecessarily increases the complexity of the decoder, thereby increasing the associated implementation cost.

[0124] In general, this disclosure describes techniques for supporting sub-picture or MCTS-based random access in video coding. More specifically, this disclosure describes an improved design for supporting mixed NAL unit types in images that are used to support sub-picture or MCTS-based random access. The description of these techniques is based on the VVC standard, but is also applicable to other video / media codec specifications.

[0125] In order to solve the above problems, the following exemplary implementations are disclosed. These implementations can be applied individually or in combination. In one example, each image is associated with an indication of whether the image includes a mixed nal_unit_type value. This indication is indicated in the PPS. This indication supports determining whether to reset the POC MSB and / or reset the DPB by marking all reference images as unused for reference. When the indication is signaled in the PPS, changes to the values ​​in the PPS can be made during merging or separate extraction. However, this is acceptable because during such code stream extraction or merging, the PPS is overwritten and replaced by other mechanisms.

[0126] Alternatively, this indication may be indicated in the tile group header, but this indication must be the same for all tile groups of a picture. However, in this case, the value may need to be changed during the substream extraction process of the MCTS / sub-image sequence. Alternatively, this indication may be indicated in the NAL unit header, but this indication must be the same for all tile groups of a picture. However, in this case, the value may need to be changed during the substream extraction process of the MCTS / sub-image sequence. Alternatively, this indication may be indicated by defining such other VCL NAL unit types, and when used for a picture, all VCL NAL units of the picture need to have the same NAL unit type value. However, in this case, the NAL unit type value of the VCL NAL units may need to be changed during the substream extraction process of the MCTS / sub-image sequence. Alternatively, this indication may be indicated by defining these other IRAPVCL NAL unit types, and when used for a picture, all VCL NAL units of the picture shall have the same NAL unit type value. However, in this case, the NAL unit type value of the VCL NAL units may need to be changed during the substream extraction process of the MCTS / sub-image sequence. Alternatively, each picture having at least one VCL NAL unit with any IRAP NAL unit type may be associated with an indication of whether the picture includes a mixed NAL unit type value.

[0127] In addition, constraints may be applied such that mixing of nal_unit_type values ​​in a picture is allowed in a limited way by only allowing mixing of IRAP and non-IRAP NAL unit types. For any particular picture, all VCL NAL units have the same NAL unit type, or some VCL NAL units have a specific IRAP NAL unit type and the rest have a specific non-IRAP VCL NAL unit type. In other words, the VCL NAL units of any particular picture cannot have multiple IRAP NAL unit types, nor can they have multiple non-IRAP NAL unit types. A picture can be considered an IRAP picture only if the picture does not include mixed nal_unit_type values ​​and the VCL NAL units have IRAP NAL unit types. The POC MSB may not be reset for any IRAP NAL unit (including IDR) that does not belong to an IRAP picture. The DPB is not reset for any IRAP NAL unit (including IDR) that does not belong to an IRAP picture, so all reference pictures are not marked as unused for reference. If at least one VCL NAL unit of a picture is an IRAP NAL unit, the TemporalId of the picture may be set to 0.

[0128] The following is a specific implementation of one or more of the above aspects. An IRAP picture can be defined as a decoded picture, the value of mixed_nalu_types_in_pic_flag is equal to 0, and the nal_unit_type of each VCL NAL unit is in the range of IDR_W_RADL to RSV_IRAP_VCL13 (including the end value). The syntax and semantics of the exemplary PPS are as follows:

[0129] pic_parameter_set_rbsp(){ Descriptors pps_pic_parameter_set_id ue(v) pps_seq_parameter_set_id ue(v) mixed_nalu_types_in_pic_flag ue(v) single_tile_in_pic_flag u(1) ...

[0130] mixed_nalu_types_in_pic_flag is set to 0, indicating that each picture of the referenced PPS has multiple VCL NAL units, and the nal_unit_type values ​​of these NAL units are different. mixed_nalu_types_in_pic_flag is set to 0, indicating that the VCL NAL units of each picture of the referenced PPS have the same nal_unit_type value.

[0131] An example block group / strip header syntax is as follows:

[0132]

[0133]

[0134] The semantics of an exemplary NAL unit header are as follows: For the VCL NAL units of any particular picture, one of the following two conditions needs to be met. All VCL NAL units have the same nal_unit_type value. Some VCL NAL units have specific IRAP NAL unit type values ​​(i.e., values ​​of nal_unit_type in the range of IDR_W_RADL to RSV_IRAP_VCL13, inclusive), while all other VCL NAL units have specific non-IRAP VCL NAL unit types (i.e., values ​​of nal_unit_type in the range of TRAIL_NUT to RSV_VCL_7, inclusive, or in the range of RSV_VCL14 to RSV_VCL15, inclusive). nuh_temporal_id_plus1–1 represents the temporal identifier of the NAL unit. The value of nuh_temporal_id_plus1 shall not be equal to 0.

[0135] The variable TemporalId is derived as follows:

[0136] TemporalId = nuh_temporal_id_plus1 – 1 (7-1)

[0137] When nal_unit_type is in the range of IDR_W_RADL to RSV_IRAP_VCL13 (inclusive), for the VCL NAL units of a picture, the TemporalId of all VCL NAL units of the picture needs to be equal to 0, regardless of the nal_unit_type values ​​of other VCL NAL units of the picture. The TemporalId value of all VCL NAL units in the same access unit must be the same. The TemporalId value of a decoded picture or access unit is the TemporalId value of the VCL NAL unit of the decoded picture or access unit.

[0138] An exemplary decoding process for a decoded image is as follows: For the current image CurrPic, the decoding process is as follows: The decoding of NAL units is described herein. The following decoding process uses syntax elements in and above the partition group header layer. Variables and functions related to image sequence numbers are derived as specified herein. The above decoding process is only called for the first partition group / slice of the image. At the beginning of the decoding process of each partition group / slice, the decoding process for reference image list construction is called to derive reference image list 0 (RefPicList[0]) and reference image list 1 (RefPicList[1]). If the current image is an IDR image, the decoding process for reference image list construction can be called for the purpose of code stream consistency checking, but it may not be necessary for the current image or images that follow the current image in decoding order.

[0139] The decoding process for reference picture list construction is as follows: the decoding process is called at the beginning of the decoding process for each partition group. The reference image is addressed by a reference index. The reference index is an index into the reference picture list. When decoding an I partition group, the reference picture list is not used to decode the partition group data. When decoding a P partition group, only reference picture list 0 (i.e., RefPicList[0]) is used to decode the partition group data. When decoding a B partition group, reference picture list 0 and reference picture list 1 (RefPicList[1]) are used to decode the partition group data. At the beginning of the decoding process for each partition group, reference picture lists RefPicList[0] and RefPicList[1] are derived. The reference picture list is used to mark reference pictures or to decode partition group data. For any partition group of an IDR picture or an I partition group of a non-IDR picture, RefPicList[0] and RefPicList[1] can be derived for the purpose of code stream consistency checking, but are not necessary for decoding the current picture or pictures that follow the current picture in decoding order. For P partition groups, RefPicList[1] can be derived for the purpose of codestream consistency checking, but is not necessary for decoding the current picture or pictures that follow the current picture in decoding order.

[0140] Figure 88 is a schematic diagram of an exemplary video decoding device 800. The video decoding device 800 is suitable for implementing the disclosed examples / embodiments described herein. The video decoding device 800 includes a downstream port 820, an upstream port 850 and / or a transceiver unit (Tx / Rx) 810. The transceiver unit 810 includes a transmitter and / or a receiver for communicating data in the upstream and / or downstream through a network. The video decoding device 800 also includes: a processor 830, including a logic unit and / or a central processing unit (CPU) for processing data; and a memory 832 for storing data. The video decoding device 800 may also include an electrical component, an optical-to-electrical (OE) component, an electrical-to-optical (EO) component, and / or a wireless communication component coupled to the upstream port 850 and / or the downstream port 820 for communicating data through an electrical communication network, an optical communication network, or a wireless communication network. The video decoding device 800 may also include an input and / or output (I / O) device 860 for communicating data with a user. The I / O device 860 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 860 may also include an input device such as a keyboard, a mouse, a trackball, etc. and / or a corresponding interface for interacting with the above-mentioned output devices.

[0141] The processor 830 is implemented by hardware and software. The processor 830 can be implemented as one or more CPU chips, one or more cores (for example, as a multi-core processor), one or more field-programmable gate arrays (FPGA), one or more application-specific integrated circuits (ASIC), and one or more digital signal processors (DSP). The processor 830 communicates with the downstream port 820, Tx / Rx 810, the upstream port 850, and the memory 832. The processor 830 includes a decoding module 814. The decoding module 814 implements the disclosed embodiments described herein, such as method 100, method 900, and / or method 1000, which can use CVS 500, VR image SHIPIN600, and / or code stream 700. The decoding module 814 can also implement any other method / mechanism described herein. In addition, the decoding module 814 can implement the codec system 200, the encoder 300, and / or the decoder 400. For example, the decoding module 814 can set a flag in the PPS to indicate when a picture includes IRAP and non-IRAP NAL units, and limit these pictures to only include a single type of IRAP NAL unit and a single type of non-IRAP NAL unit. Therefore, the decoding module 814 enables the video decoding device 800 to provide additional functionality and / or decoding efficiency when decoding video data. Therefore, the decoding module 814 improves the functionality of the video decoding device 800 and solves problems unique to the field of video decoding. In addition, the decoding module 814 affects the transition of the video decoding device 800 to different states. Alternatively, the decoding module 814 can be implemented as instructions stored in the memory 832 and executed by the processor 830 (for example, implemented as a computer program product stored in a non-transitory medium).

[0142] The memory 832 includes one or more memory types, such as a disk, a tape drive, a solid-state hard disk, a read only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 832 can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the program.

[0143] Fig. 9The present invention is a flowchart of an exemplary method 900 for encoding a video sequence such as CVS 500 including an image having a mixed NAL unit type into a bitstream such as bitstream 700, wherein the bitstream 700 includes a VR image video stream 600 merged from a plurality of sub-image video streams 601 to 603 of a plurality of video resolutions. The method 900 may be performed by an encoder such as the encoding and decoding system 200, the encoder 300 and / or the video decoding device 800 when performing the method 100.

[0144] Method 900 may start with: an encoder receives a video sequence including multiple VR images and other images, and determines to encode the video sequence into a bitstream based on user input, etc. In step 901, the encoder determines whether the current image includes multiple sub-images of different types. These types may include at least one slice of an image having a portion of an IRAP sub-image and at least one slice of an image having a portion of a non-IRAP NAL sub-image. In step 903, the encoder encodes the slices of the sub-images of the image into multiple VCL NAL units in the bitstream. These VCL NAL units may include one or more IRAP NAL units and one or more non-IRAP NAL units. For example, the encoding step may include merging sub-codestreams of different resolutions into a single codestream to be sent to a decoder.

[0145] In step 905, the encoder encodes the PPS into the codestream and encodes the flag into the PPS in the codestream. In a specific example, encoding the PPS may include changing a previously encoded PPS to include a flag value, for example in response to merging sub-codestreams. When the NAL unit type values ​​of all VCL NAL units associated with the image are the same, the flag may be set to a first value. When the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including one or more sub-images in the image, the flag may also be set to a second value. For example, the first NAL unit type value may indicate that the image includes an IRAP sub-image, and the second NAL unit type value may indicate that the image also includes a non-IRAP sub-image. In addition, the first NAL unit type value may be equal to one of IDR_W_RADL, IDR_N_LP, or CRA_NUT. In addition, the second NAL unit type value may be equal to one of TRAIL_NUT, RADL_NUT, or RASL_NUT. In a specific example, the flag may be mixed_nalu_types_in_pic_flag. In a specific example, mixed_nalu_types_in_pic_flag may be set to 1 to indicate that each image with reference to the PPS including the flag has multiple VCL NAL units. In addition, the flag indicates that the VCL NAL units associated with the corresponding image do not all have the same NAL unit type (nal_unit_type) value. In another specific example, mixed_nalu_types_in_pic_flag may be set to 0 to indicate that each image with reference to the PPS including the flag has one or more VCL NAL units. In addition, the flag indicates that all VCL NAL units of the corresponding image have the same nal_unit_type value.

[0146] In step 907, the encoder may store the code stream to send to the decoder.

[0147] Fig.10 A flowchart of an exemplary method 1000 for decoding a video sequence such as CVS 500 including an image with mixed NAL unit types from a bitstream such as bitstream 700, wherein the bitstream 700 includes a VR image video stream 600 merged from a plurality of sub-image video streams 601 to 603 of a plurality of video resolutions. The method 1000 may be performed by a decoder such as the codec system 200, the decoder 400, and / or the video decoding device 800 when executing the method 100.

[0148] When the decoder begins to receive a bitstream representing decoded data of a video sequence, for example as a result of method 900, method 1000 is performed. In step 1001, the decoder receives the bitstream. The bitstream includes a plurality of sub-images and a flag associated with an image. In a specific example, the bitstream may include a PPS, and the PPS includes a flag. In addition, the sub-image is included in a plurality of VCL NAL units. For example, a slice associated with a sub-image is included in a VCL NAL unit.

[0149] In step 1003, when the flag is set to a first value, the decoder determines that the NAL unit type values ​​of all VCL NAL units associated with the image are the same. In addition, when the flag is set to a second value, the decoder determines that the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including one or more sub-images in the image. For example, the first NAL unit type value may indicate that the image includes an IRAP sub-image, and the second NAL unit type value may indicate that the image also includes a non-IRAP sub-image. In addition, the first NAL unit type value may be equal to one of IDR_W_RADL, IDR_N_LP, or CRA_NUT. In addition, the second NAL unit type value may be equal to one of TRAIL_NUT, RADL_NUT, or RASL_NUT. In a specific example, the flag may be mixed_nalu_types_in_pic_flag. When each picture representing the reference PPS has multiple VCL NAL units and the VCL NAL units do not have the same NAL unit type (nal_unit_type) value, mixed_nalu_types_in_pic_flag may be set to 1. In addition, when each picture representing the reference PPS has one or more VCL NAL units and the VCL NAL units of each picture of the reference PPS have the same nal_unit_type value, mixed_nalu_types_in_pic_flag may be set to 0.

[0150] In step 1005, the decoder may decode the one or more sub-pictures according to the NAL unit type value. In step 1007, the decoder may also forward the one or more sub-pictures for display as part of the decoded video sequence.

[0151] Fig.11The present invention is a schematic diagram of an exemplary system 1100 for encoding a video sequence such as CVS 500 including images with mixed NAL unit types into a bitstream such as bitstream 700, wherein the bitstream 700 includes a VR image video stream 600 merged from a plurality of sub-image video streams 601 to 603 of a plurality of video resolutions. The system 1100 may be implemented by an encoder and a decoder such as the encoding and decoding system 200, the encoder 300, the decoder 400 and / or the video decoding device 800. In addition, the system 1100 may be used to implement the method 100, the method 900 and / or the method 1000.

[0152] The system 1100 includes a video encoder 1102. The video encoder 1102 includes a determination module 1101 for determining whether an image includes a plurality of sub-images of different types. The video encoder 1102 also includes an encoding module 1103 for encoding the sub-images of the image into a plurality of VCL NAL units in a bitstream. The encoding module 1103 is also used to encode a flag into the bitstream, wherein when the NAL unit type values ​​of all VCL NAL units associated with the image are the same, the flag is set to a first value, and when the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image, the flag is set to a second value. The video encoder 1102 also includes a storage module 1105 for storing the bitstream to be sent to a decoder. The video encoder 1102 also includes a sending module 1107 for sending the bitstream to a video decoder 1110. The video encoder 1102 can also be used to perform any step of the method 900.

[0153] The system 1100 further includes a video decoder 1110. The video decoder 1110 includes a receiving module 1111 for receiving a bitstream, the bitstream including a plurality of sub-images and a flag associated with an image, wherein the sub-images are included in a plurality of VCL NAL units. The video decoder 1110 further includes a determining module 1113 for determining that the NAL unit type values ​​of all VCL NAL units associated with the image are the same when the flag is set to a first value. The determining module 1113 is further configured to determine that the first NAL unit type value of the VCL NAL unit including one or more sub-images in the image is different from the second NAL unit type value of the VCL NAL unit including the one or more sub-images in the image when the flag is set to a second value. The video decoder 1110 further includes a decoding module 1115 for decoding the one or more sub-images according to the NAL unit type value. The video decoder 1110 further includes a forwarding module 1117 for forwarding the one or more sub-images displayed as part of a decoded video sequence. The video decoder 1110 may also be configured to perform any step of the method 1000.

[0154] A first component is directly coupled to a second component when there is no other intermediate component between the first component and the second component except a line, trace or other medium. A first component is indirectly coupled to a second component when there is another intermediate component other than a line, trace or other medium between the first component and the second component. The term "coupled" and its variations include direct coupling and indirect coupling. Unless otherwise specified, the term "approximately" is used to refer to ±10% of the number described below.

[0155] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and some steps may be omitted or combined.

[0156] Although the present invention provides a plurality of specific embodiments, it should be understood that the disclosed systems and methods may also be embodied in a variety of other specific forms without departing from the spirit or scope of the present invention. The examples of the present invention should be considered illustrative rather than restrictive, and the present invention is not limited to the details given in this text. For example, various elements or components may be combined or merged in another system, or certain features may be omitted or not implemented.

[0157] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, components, techniques or methods. Other changes, substitutions and replacement examples are obvious to those skilled in the art, and all do not depart from the spirit and scope disclosed herein.

Claims

1. A method of encoding, characterized in that: The method comprises: Encoding multiple images into multiple video coding layer (VCL) network abstraction layer (NAL) units of a bitstream; encoding a flag into the code stream; When it is determined that the NAL unit type values ​​of all VCL NAL units associated with a picture are the same, the flag is set to a first value; The flag is set to a second value when it is determined that one or more VCL NAL units in the one image have a first NAL unit type value, and all other VCL NAL units in the one image except the one or more VCL NAL units have a second NAL unit type value; the first NAL unit type value is different from the second NAL unit type value.

2. The method according to claim 1, characterized in that The code stream includes a picture parameter set PPS, and the flag is encoded in the PPS.

3. The method according to claim 1 or 2, characterized in that: The first NAL unit type is an intra random access point (IRAP) type, and the second NAL unit type is a non-IRAP type.

4. The method according to claim 1 or 2, characterized in that: The first NAL unit type value is equal to an immediate decoding refresh IDR with a decodable random access preceding picture (IDR_W_RADL), an IDR without preceding pictures (IDR_N_LP), or a pure random access CRANAL unit type (CRA_NUT).

5. The method according to claim 1 or 2, characterized in that: The second NAL unit type value is equal to the trailing picture NAL unit type (TRAIL_NUT).

6. The method according to claim 1 or 2, characterized in that: The flag is mixed_nalu_types_in_pic_flag.

7. The method according to claim 6, characterized in that When the mixed_nalu_types_in_pic_flag is equal to 1, it indicates that the picture of the reference picture parameter set (PPS) has multiple VCL NAL units and the VCL NAL units do not have the same NAL unit type (nal_unit_type) value; When the mixed_nalu_types_in_pic_flag is equal to 0, it indicates that the picture referring to the PPS has one or more VCL NAL units and the VCL NAL units have the same nal_unit_type value.

8. A video decoding device, characterized in that: include: A processor, and a memory coupled to the processor, wherein the processor is configured to execute the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable medium, characterized in that The non-transitory computer-readable medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the video decoding device performs the method according to any one of claims 1 to 7.

10. An encoder, characterized in that: include: An encoding module, for encoding a plurality of images into a plurality of video coding layer (VCL) network abstraction layer (NAL) units of a bitstream; and encoding the flag into the code stream; Identify modules for: When it is determined that the NAL unit type values ​​of all VCL NAL units associated with a picture are the same, the flag is set to a first value; When it is determined that one or more VCL NAL units in the one image have a first NAL unit type value, and other VCL NAL units in the one image except the one or more VCL NAL units have a second NAL unit type value, the flag is set to a second value; the first NAL unit type value is different from the second NAL unit type value.

11. The encoder according to claim 10, characterized in that The encoder is further configured to perform the method according to any one of claims 2 to 7.

12. A device for encoding a code stream, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store at least one of a code stream or data obtained by decoding the code stream; The processor is used to execute program instructions to perform the encoding method according to any one of claims 1 to 7.

13. A computer program product, characterized in that The invention comprises computer program instructions, which, when executed by one or more processors, enable an encoder to perform the encoding method according to any one of claims 1 to 7.

14. A decoding method, characterized in that: include: Receiving a bitstream, the bitstream comprising coded data of one or more images and a flag, the coded data of the one or more images being in a plurality of video coding layer (VCL) network abstraction layer (NAL) units; Decoding the bitstream to obtain the flag, wherein the flag is a first value indicating that the first NAL unit type value of all VCL NAL units associated with an image is the same; the flag is a second value indicating that one or more VCL NAL units in the image have the first NAL unit type value, and other VCL NAL units in the image except the one or more VCL NAL units have the second NAL unit type value, and the first NAL unit type value is different from the second NAL unit type value; The one or more images are decoded according to the flag.

15. A decoding device, characterized in that: include: A receiving unit, configured to receive a code stream, wherein the code stream includes coded data of one or more images and a flag, wherein the coded data of the one or more images are in a plurality of video coding layer (VCL) network abstraction layer (NAL) units; A decoding unit, configured to decode the bitstream to obtain the flag, wherein the flag is a first value indicating that the first NAL unit type value of all VCL NAL units associated with an image is the same; the flag is a second value indicating that one or more VCL NAL units in the image have the first NAL unit type value, and other VCL NAL units in the image except the one or more VCL NAL units have a second NAL unit type value, and the first NAL unit type value is different from the second NAL unit type value; The decoding unit is further configured to decode the one or more images according to the flag.

16. A device for storing a code stream, characterized in that: including a receiver and a memory, The receiver is used to receive one or more code streams, the code streams including coded data and flags of multiple images, the multiple images are encoded in multiple video coding layer (VCL) network abstraction layer (NAL) units, when the flag is a first value, it indicates that the first NAL unit type value of all VCL NAL units associated with one image is the same; when the flag is a second value, it indicates that the one or more VCL NAL units in the one image have the first NAL unit type value, and the other VCL NAL units in the one image except the one or more VCL NAL units have the second NAL unit type value, and the first NAL unit type value is different from the second NAL unit type value; The memory is used to store the one or more code streams.

17. The device for storing code stream according to claim 16, characterized in that: The code stream includes a picture parameter set PPS, and the PPS includes the flag.

18. The device for storing code stream according to claim 16 or 17, characterized in that: The first NAL unit type value is equal to an immediate decoding refresh IDR with a decodable random access preceding picture (IDR_W_RADL), an IDR without preceding pictures (IDR_N_LP), or a pure random access CRANAL unit type (CRA_NUT).

19. The device for storing code stream according to claim 16 or 17, characterized in that: The second NAL unit type value is equal to the trailing picture NAL unit type (TRAIL_NUT).

20. The device for storing code stream according to claim 16 or 17, characterized in that: The flag is mixed_nalu_types_in_pic_flag.

21. The device for storing code stream according to claim 20, characterized in that: When the mixed_nalu_types_in_pic_flag is equal to 1, it indicates that the picture of the reference picture parameter set (PPS) has multiple VCL NAL units and the VCL NAL units do not have the same NAL unit type (nal_unit_type) value; When the mixed_nalu_types_in_pic_flag is equal to 0, it indicates that the picture referring to the PPS has one or more VCL NAL units and the VCL NAL units have the same nal_unit_type value.

22. The device for storing code stream according to claim 16 or 17, characterized in that: The system also includes a processor configured to select a target code stream from one or more code streams stored in the memory, and transmit the target code stream to a destination device.

Images