Video coding bitstream extraction using indicator

Abstract

A video coding mechanism is disclosed. The mechanism includes receiving an extracted bitstream, the extracted bitstream being a bitstream extracted from an input bitstream containing a set of sub-pictures by a sub-bitstream extraction process. The extracted bitstream only contains a subset of the sub-pictures of the input bitstream for the sub-bitstream extraction process. A flag in the extracted bitstream is set to indicate that there is sub-picture information related to the subset of the sub-pictures in the extracted bitstream. One or more sub-picture identifiers (IDs) of the subset of the sub-pictures are obtained according to the flag. The subset of the sub-pictures is decoded according to the sub-picture IDs.

Description

[0001] This application is a divisional application. The original application has the application number 202080045816.X and the original application date is June 15, 2020. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This invention generally relates to video decoding, and more specifically to bitstream extraction in video decoding. Background Technology

[0003] Even short videos require a significant amount of video data to describe, which can be challenging when streaming or otherwise transmitting data over bandwidth-constrained communication networks. Therefore, video data is typically compressed before transmission over modern telecommunications networks. Due to limited memory resources, the size of the video must be considered when storing it on storage devices. Video compression devices typically encode video data using software and / or hardware at the source side before transmitting or storing it, reducing the amount of data required to represent the digital video image. Video decompression devices then decode the video data and receive the compressed data at the destination side. Given limited network resources and the growing demand for higher video quality, there is a need to improve compression and decompression techniques to increase compression ratios with minimal impact on image quality. Summary of the Invention

[0004] In one embodiment, the present invention includes a method implemented in a decoder, the method comprising: a receiver of the decoder receiving an extracted bitstream, the extracted bitstream being extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, the extracted bitstream containing only a subset of the sub-images of the input bitstream used in the sub-bitstream extraction process; a processor of the decoder determining a flag in the extracted bitstream, the flag being set to indicate the presence of sub-image information in the extracted bitstream associated with the subset of the sub-images; the processor obtaining one or more sub-image identifiers (IDs) of the subset of the sub-images based on the flags; and the processor decoding the subset of the sub-images based on the sub-image IDs.

[0005] Some video decoding sequences may include images encoded as sets of sub-images. Sub-images can be associated with a sub-image ID, which can be used to represent the position of the sub-image relative to the image. In some cases, this sub-image information can be inferred. In such cases, this sub-image information can be excluded from the bitstream to improve decoding efficiency. Some processes can extract sub-bitstreams from the bitstream for independent transmission to the end user. In this case, the sub-bitstream contains only a subset of the sub-images contained in the original bitstream. While sub-image information can be inferred when all sub-images are present, the decoder cannot make this inference when only a subset of sub-images is present. This example includes mechanisms to prevent decoding errors during sub-bitstream extraction. Specifically, if a sub-bitstream is extracted from the bitstream, the encoder and / or splitter includes sub-image information representing at least a subset of the sub-images in the sub-bitstream. Furthermore, the encoder / splitter includes a flag indicating that sub-image information is included in the sub-bitstream. The decoder can read this flag, obtain the correct sub-image information, and decode the sub-bitstream. Therefore, the disclosed mechanism enables the encoder and / or decoder to have additional functionality, thereby avoiding errors. Furthermore, the disclosed mechanism can improve decoding efficiency by supporting sub-stream extraction instead of sending the entire stream. This can reduce the utilization of processor, memory, and / or network resources at the encoder and / or decoder.

[0006] Optionally, according to any of the above aspects, in another implementation of said aspect, the processor further includes: obtaining the bit length of the syntax element containing the one or more sub-image IDs.

[0007] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the flag, the sub-image ID, and the length are obtained from the sequence parameter set (SPS) in the extracted bitstream.

[0008] Alternatively, according to any of the above aspects, in another implementation of said aspect, the flag is a subpic_info_present_flag.

[0009] Alternatively, according to any of the above aspects, in another implementation of said aspect, the subpicture ID is contained in the SPS subpicture identifier (sps_subpic_id[i]) syntax structure.

[0010] Alternatively, according to any of the above aspects, in another implementation of said aspect, the length is contained in the SPS subpic ID length minus one plus one (sps_subpic_id_len_minus1+1) syntax structure.

[0011] Optionally, according to any of the above aspects, in another implementation of said aspect, when the extracted bitstream is extracted from the input bitstream through the sub-bitstream extraction process, the flag needs to be set to 1 to indicate that the coded layer video sequence (CLVS) contains the sub-image information, and each image of the CLVS contains multiple sub-images.

[0012] In one embodiment, the present invention includes a method implemented in an encoder, the method comprising: a processor of the encoder encoding an input bitstream containing a subset of sub-images; the processor performing a sub-bitstream extraction process on the input bitstream to create an extracted bitstream, the extracted bitstream containing only a subset of the sub-images of the input bitstream; the processor encoding one or more sub-image IDs of the subset of the sub-images in the extracted bitstream into the extracted bitstream; the processor setting a flag in the extracted bitstream to indicate the presence of sub-image information related to the subset of the sub-images in the extracted bitstream; and a memory coupled to the processor storing the bitstream for communication with a decoder.

[0013] Some video decoding sequences may include images encoded as sets of sub-images. Sub-images can be associated with a sub-image ID, which can be used to represent the position of the sub-image relative to the image. In some cases, this sub-image information can be inferred. In such cases, this sub-image information can be excluded from the bitstream to improve decoding efficiency. Some processes can extract sub-bitstreams from the bitstream for independent transmission to the end user. In this case, the sub-bitstream contains only a subset of the sub-images contained in the original bitstream. While sub-image information can be inferred when all sub-images are present, the decoder cannot make this inference when only a subset of sub-images is present. This example includes mechanisms to prevent decoding errors during sub-bitstream extraction. Specifically, if a sub-bitstream is extracted from the bitstream, the encoder and / or splitter includes sub-image information representing at least a subset of the sub-images in the sub-bitstream. Furthermore, the encoder / splitter includes a flag indicating that sub-image information is included in the sub-bitstream. The decoder can read this flag, obtain the correct sub-image information, and decode the sub-bitstream. Therefore, the disclosed mechanism enables the encoder and / or decoder to have additional functionality, thereby avoiding errors. Furthermore, the disclosed mechanism can improve decoding efficiency by supporting sub-stream extraction instead of sending the entire stream. This can reduce the utilization of processor, memory, and / or network resources at the encoder and / or decoder.

[0014] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the processor further includes encoding the bit length of the syntax element containing the one or more sub-image IDs into the extracted bitstream.

[0015] Alternatively, in any of the above aspects, in another implementation of said aspect, the flag, the sub-image ID, and the length are encoded into the SPS in the extracted bitstream.

[0016] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the flag is subpic_info_present_flag.

[0017] Alternatively, according to any of the above aspects, in another implementation of said aspect, the subpicture ID is contained in the sps_subpic_id[i] syntax structure.

[0018] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the length is contained in the sps_subpic_id_len_minus1+1 syntax structure.

[0019] Optionally, according to any of the above aspects, in another implementation of said aspect, when the extracted bitstream is extracted from the input bitstream through the sub-bitstream extraction process, the flag needs to be set to 1 to indicate that the CLVS contains the sub-image information, and each image of the CLVS contains multiple sub-images.

[0020] In one embodiment, the present invention includes a video decoding device comprising: a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method described in any of the preceding aspects.

[0021] In one embodiment, the present invention includes a non-transitory computer-readable medium comprising a computer program product for use by a video decoding device, wherein the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium, which, when executed by a processor, cause the video decoding device to perform the method according to any of the preceding aspects.

[0022] In one embodiment, the present invention includes a decoder comprising: a receiving module for receiving an extracted bitstream, the extracted bitstream being extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, the extracted bitstream containing only a subset of the sub-images of the input bitstream used in the sub-bitstream extraction process; a determining module for determining a flag in the extracted bitstream, the flag being set to indicate the presence of sub-image information in the extracted bitstream associated with the subset of the sub-images; an acquiring module for acquiring one or more sub-image IDs of the subset of the sub-images based on the flag; a decoding module for decoding the subset of the sub-images based on the sub-image IDs; and a forwarding module for forwarding the subset of the sub-images for display as part of a decoded video sequence.

[0023] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the decoder is further configured to perform the method described according to any of the foregoing aspects.

[0024] In one embodiment, the present invention includes an encoder comprising: a first encoding module for encoding an input bitstream containing a subset of sub-images; a bitstream extraction module for performing a sub-bitstream extraction process on the input bitstream to create an extracted bitstream, the extracted bitstream containing only a subset of the sub-images of the input bitstream; a second encoding module for encoding one or more sub-image IDs of the subset of the sub-images in the extracted bitstream into the extracted bitstream; a setting module for setting a flag in the extracted bitstream to indicate that there is sub-image information related to the subset of the sub-images in the extracted bitstream; and a storage module for storing the bitstream for communication with a decoder.

[0025] Alternatively, according to any of the foregoing aspects, in another implementation of said aspect, the encoder is further configured to perform the method described according to any of the foregoing aspects.

[0026] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0027] These and other features will become clearer from the following detailed description taken in conjunction with the accompanying drawings and claims. Attached Figure Description

[0028] To gain a more complete understanding of the present invention, reference is made to the following brief description in conjunction with the accompanying drawings and specific embodiments, wherein the same reference numerals denote the same parts.

[0029] Figure 1 A flowchart illustrating an exemplary method for decoding video signals.

[0030] Figure 2 This is a schematic diagram of an exemplary encoding and decoding (encoding / decoding) system used for video decoding.

[0031] Figure 3 This is a schematic diagram of an exemplary video encoder.

[0032] Figure 4 This is a schematic diagram of an exemplary video decoder.

[0033] Figure 5 This is a schematic diagram of multiple sub-image / video streams extracted from an image / video stream.

[0034] Figure 6 This is a schematic diagram of an exemplary bitstream that has been divided into sub-bitstreams.

[0035] Figure 7 This is a schematic diagram of an exemplary video decoding device.

[0036] Figure 8 A flowchart illustrating an exemplary method for encoding video sequences into a bitstream and extracting sub-bitstreams while reducing ID errors.

[0037] Figure 9 A flowchart illustrating an exemplary method for decoding a video sequence from a sub-bitstream extracted from a bitstream.

[0038] Figure 10 This is a schematic diagram of an exemplary system for decoding video sequences of images in a bitstream, extracting sub-bitstreams, and reducing ID errors. Detailed Implementation

[0039] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. The invention should not be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but can be modified within the full scope of the appended claims and their equivalents.

[0040] The following terms are defined as described below, unless used in the context to the contrary herein. Specifically, the following definitions are intended to describe the invention more clearly in different contexts. However, terms may be described differently in different contexts. Therefore, the following definitions should be considered supplementary information and should not be construed as limiting any other definitions provided herein for these terms.

[0041] A bitstream is a sequence of bits containing video data that is compressed for transmission between an encoder and a decoder. An encoder is a device that uses an encoding process to compress video data into a bitstream. A decoder is a device that uses a decoding process to reconstruct the video data from the bitstream for display. An image is an array of luminance and / or chrominance samples that create a frame or field. For clarity, the image being encoded or decoded can be referred to as the current image. A sub-image is a rectangular region in an image consisting of one or more stripes. The substream extraction process is a specified mechanism that removes Network Abstraction Layer (NAL) units from the bitstream that are not part of the target set, resulting in an output substream that includes the NAL units included in the target set. An NAL unit is a syntax structure containing data bytes and a representation of the data type they contain. NAL units include Video Coding Layer (VCL) NAL units containing video data and non-VCL NAL units containing supporting syntax data. The input bitstream is a bitstream containing a complete set of NAL units before the substream extraction process is applied. The extracted bitstream, also known as a sub-bitstream, is the bitstream output from the bitstream extraction process and includes a subset of the NAL units from the input bitstream. A set is a collection of different items. A subset is a collection of items such that every item in the subset is included in the set, and at least one item in the set is excluded from the subset. Sub-image information is any data that describes a sub-image. A flag is a data structure containing a sequence of bits that can be set to represent the corresponding data. A sub-image identifier (ID) is a data item that uniquely identifies the corresponding sub-image. The length of a data structure is the number of bits contained in the data structure. A coded layer video sequence (CLVS) is a sequence of coded video data containing one or more layers of images. When a CLVS contains a single layer or when a CLVS is discussed outside of a layer-specific context, it can be called a coded video sequence (CVS). A sequence parameter set (SPS) is a set of parameters containing data associated with the image sequence. A decoded video sequence is an image sequence reconstructed by a decoder and prepared for display to the user.

[0042] This paper uses the following abbreviations: Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Coded Video Sequence (CVS), Joint Video Experts Team (JVET), Motion-Constrained Tile Set (MCTS), Maximum Transfer Unit (MTU), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sequence Parameter Set (SPS), Sub-Picture Unit (SPU), Versatile Video Coding (VVC), and Working Draft (WD).

[0043] Many video compression techniques can be used to reduce the size of video files while minimizing data loss. For example, video compression techniques may include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in a video sequence. For block-based video decoding, video strips (e.g., video images or portions of video images) can be segmented into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-frame coded (I) strip of an image are encoded using spatial prediction against reference samples in neighboring blocks within the same image. Video blocks in an inter-frame unidirectional prediction (P) strip or bidirectional prediction (B) strip of an image may use spatial prediction against reference samples in neighboring blocks within the same image, or use temporal prediction against reference samples in other reference images. An image may be called a frame, and a reference image may be called a reference frame. Spatial or temporal prediction produces a prediction block representing an image block. Residual data represents the pixel difference between the original image block and the prediction block. Therefore, inter-frame coded blocks are encoded based on the motion vectors of the blocks pointing to the reference samples constituting the prediction blocks and residual data representing the difference between the coded blocks and the prediction blocks. Intra-frame coded blocks are encoded based on the intra-frame coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to generate residual transform coefficients, which can be quantized. The quantized transform coefficients are initially arranged in a two-dimensional array. The quantized transform coefficients can be scanned to generate a one-dimensional vector of transform coefficients. Entropy decoding can be applied to achieve further compression. This video compression technique is described in detail below.

[0044] To ensure correct decoding of encoded video, the video is encoded and decoded according to the corresponding video coding standards. These standards include ITU-T H.261, ISO / IEC MPEG-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes scalable video coding (SVC), multiview video coding (MVC), multiview video coding plus depth (MVC+D), and extended versions such as three-dimensional AVC (3D-AVC). HEVC includes extended versions such as scalable HEVC (SHVC), multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The Joint Video Experts Team (JVET) of ITU-T and ISO / IEC has begun developing a video coding standard called versatile video coding (VVC). VVC is included in the working draft (WD), which includes JVET-N1001-v8.

[0045] Video decoding sequences consist of image sequences. In some cases, these images can be further segmented into sets of sub-images, where each sub-image comprises a different region of the image. Sub-images can support different spatial portions of the image being processed differently in the decoder. For example, in a virtual reality (VR) context, only a portion of the entire image is shown to the user. Therefore, sub-images can be used to send different parts of the image to the decoder at different resolutions and / or even omit certain parts of the image. This can improve decoding efficiency. In another example, a teleconference application can dynamically increase the size and / or resolution of the image of a participant who is actively speaking and decrease the size / resolution of that participant's image when that participant stops speaking. Including each participant in a different sub-image supports this dynamic change associated with one participant without affecting images associated with other participants. Sub-images can be associated with a sub-image ID. The sub-image ID uniquely identifies the corresponding sub-image. Therefore, the sub-image ID can be used to indicate the position of the sub-image relative to the image and / or to perform changes in the sub-image-level decoding process. In some cases, sub-image information, such as the sub-image ID, can be inferred. For example, when an image comprises nine sub-images, the decoder can infer the sub-image IDs from 0 to 8. In this case, such sub-image information can be excluded from the bitstream to improve decoding efficiency.

[0046] However, some processes can be used to extract sub-streams from the original bitstream for independent transmission to the end user. In this case, the sub-stream contains only a subset of the sub-images contained in the original bitstream. While sub-image information can be inferred when all sub-images are present, the decoder cannot make this inference when only a subset of sub-images exists. For example, the encoder might send only sub-image 3 out of 9 sub-images and sub-image 4 out of 9 sub-images to the decoder. If sub-image information is omitted, the decoder may not be able to determine which sub-images it is receiving and how they should be displayed. In this case, the bitstream is considered consistent because missing data related to the bitstream can be inferred. However, the extracted sub-stream is inconsistent because some missing data related to the sub-stream cannot be inferred.

[0047] This paper discloses a mechanism to prevent decoding errors during sub-stream extraction. Specifically, when extracting a sub-stream from the bitstream, the encoder and / or striper encodes sub-image information—at least a subset of sub-images—into a parameter set within the sub-stream. Furthermore, the encoder / striper includes a flag indicating that sub-image information is included in the sub-stream. The decoder can read this flag, obtain the correct sub-image information, and decode the sub-stream. This sub-image information may include a sub-image ID in a syntax element and a length data element representing the bit length of the sub-image ID syntax element. Therefore, the disclosed mechanism provides the encoder and / or decoder with additional functionality to avoid sub-image-related decoding errors. Moreover, the disclosed mechanism improves decoding efficiency by supporting sub-stream extraction instead of sending the entire bitstream. This reduces the utilization of processor, memory, and / or network resources at the encoder and / or decoder.

[0048] Figure 1 This is a flowchart of an exemplary method 100 for decoding a video signal. Specifically, the video signal is encoded at the encoder side. The encoding process compresses the video signal using various mechanisms to reduce the video file size. A smaller file size facilitates sending the compressed video file to the user while reducing associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process typically corresponds to the encoding process to ensure consistent reconstruction of the video signal by the decoder.

[0049] In step 101, a video signal is input to an encoder. For example, the video signal may be an uncompressed video file stored in memory. In another example, the video file may be captured by a video capture device (e.g., a camera) and encoded to support live video streaming. The video file may include audio and video components. The video components contain a series of image frames that, when viewed sequentially, produce a visual effect of motion. These frames contain pixels represented according to light (referred herein to as the luminance component (or luminance sample)) and color (referred to as the chrominance component (or chrominance sample)). In some examples, these frames may also contain depth values ​​to support three-dimensional viewing.

[0050] In step 103, the video is segmented into blocks. Segmentation involves subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), frames can first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64×64 pixels). A CTU contains luma and chroma samples. CTUs can be divided into blocks using a coding tree, and these blocks can then be recursively subdivided until a configuration supporting further coding is obtained. For example, the luma component of a frame can be subdivided until the blocks contain relatively uniform luma values. Similarly, the chroma component of a frame can be subdivided until the blocks contain relatively uniform color values. Therefore, the segmentation mechanism varies depending on the content of the video frame.

[0051] In step 105, various compression mechanisms are used to compress the image patches segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction aims to take advantage of the fact that objects in common scenes tend to appear in consecutive frames. Therefore, it is not necessary to repeatedly describe the patches of objects depicted in the reference frame in adjacent frames. Specifically, an object (such as a table) can remain in a constant position across multiple frames. Therefore, the table is described only once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects in multiple frames. Furthermore, moving objects can be represented across multiple frames due to reasons such as object movement or camera movement. In a specific example, video can be displayed over multiple frames as a moving car on the screen. Motion vectors can be used to describe this movement. A motion vector is a two-dimensional vector that provides the offset from the coordinates of an object in a frame to the coordinates of that object in a reference frame. Therefore, inter-frame prediction can encode image patches in the current frame as a set of motion vectors representing the offset relative to the corresponding patch in the reference frame.

[0052] Intra-frame prediction encodes blocks within a common frame. Intra-frame prediction leverages the fact that luma and chroma components tend to cluster within a frame. For example, a patch of green in a section of a tree is often adjacent to several similar patches of green. Intra-frame prediction uses various directional prediction modes (e.g., 33 modes in HEVC), planar modes, and direct current (DC) modes. A directional mode indicates that the current block is similar / identical to samples of neighboring blocks in the corresponding direction. A planar mode indicates that a series of blocks in a row / column (e.g., a plane) can be interpolated based on neighboring blocks at row edges. In effect, a planar mode represents a smooth transition of light / color along a row / column using a relatively constant slope of the changing values. The DC mode is used for boundary smoothing, indicating that a block is similar / identical to the average of samples associated with all neighboring blocks, which are associated with the angular direction of the directional prediction mode. Therefore, intra-frame predicted blocks can represent image blocks as values ​​of various relational prediction modes rather than actual values. Furthermore, inter-frame predicted blocks can represent image blocks as motion vector values ​​rather than actual values. In either case, the predicted block may not accurately represent the image block in some situations. All differences are stored in residual blocks. The residual blocks can be transformed to further compress the file.

[0053] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to the in-loop filtering scheme. The block-based prediction described above may enable the creation of block-like images on the decoder side. Furthermore, the block-based prediction scheme can encode blocks and then reconstruct the encoded blocks for later use as reference blocks. The in-loop filtering scheme iteratively applies noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to blocks / frames. These filters reduce such block artifacts, allowing for accurate reconstruction of the encoded file. Furthermore, these filters reduce artifacts in the reconstructed reference block, making it less likely that artifacts will generate other artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0054] In step 109, once the video signal has been segmented, compressed, and filtered, the resulting data is encoded into a bitstream. The bitstream includes the aforementioned data as well as any signal data required to support proper video signal reconstruction at the decoder side. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in memory for transmission to the decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 can be performed consecutively and / or simultaneously across multiple frames and blocks. Figure 1The order shown is presented for clarity and ease of description and is not intended to restrict the video decoding process to a specific order.

[0055] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the bitstream to determine the segmented portions of a frame. The segmentation should match the block segmentation result in step 103. The entropy encoding / entropy decoding used in step 111 is described here. The encoder makes many choices during compression, such as selecting a block segmentation scheme from multiple possible options based on the spatial location of values ​​in one or more input images. Indicating the exact choice may use a large number of bits. Bits used herein are binary values ​​treated as variables (e.g., bit values ​​that can vary depending on the context). Entropy decoding allows the encoder to discard any options that are obviously unsuitable for a particular situation, leaving a set of usable options. A codeword is then assigned to each usable option. The length of the codeword depends on the number of usable options (e.g., one bit for two options, two bits for three or four options, etc.). The encoder then encodes the codeword for the selected option. This scheme reduces the codeword size because the codeword size is about the same as the codeword required to uniquely represent an option from a small subset of the available options, rather than uniquely representing an option from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of available options in a similar manner to the encoder. By determining the set of available options, the decoder can read the codeword and determine the selection made by the encoder.

[0056] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate residual blocks. Then, the decoder uses the residual blocks and corresponding prediction blocks to reconstruct image blocks based on the segmentation. The prediction blocks may include intra-frame prediction blocks and inter-frame prediction blocks generated on the encoder side in step 105. Then, the reconstructed image blocks are located in frames of the reconstructed video signal based on the segmentation data determined in step 111. The syntax of step 113 can also be indicated in the bitstream by the entropy decoding described above.

[0057] In step 115, the frames of the reconstructed video signal are filtered on the encoder side in a manner similar to that in step 107. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters can be applied to the frames to remove block artifacts. Once the frames have been filtered, in step 117, the video signal can be output to a display for viewing by the end user.

[0058] Figure 2This is a schematic diagram of an exemplary encoding and decoding (encoding / decoding) system 200 for video decoding. Specifically, the encoding / decoding system 200 is capable of implementing operation method 100. The encoding / decoding system 200 broadly describes the components used in the encoder and decoder. The encoding / decoding system 200 receives a video signal and segments the video signal, as described in steps 101 and 103 of operation method 100, thereby generating a segmented video signal 201. Then, when acting as an encoder, the encoding / decoding system 200 compresses the segmented video signal 201 into an encoded bitstream, as described in steps 105, 107, and 109 of method 100. When acting as a decoder, the encoding / decoding system 200 generates an output video signal from the bitstream, as described in steps 111, 113, 115, and 117 of operation method 100. The encoding / decoding system 200 includes a general decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control and analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown in the figure. Figure 2 In the diagram, black lines represent the movement of data to be encoded / decoded, while dashed lines represent the movement of control data controlling the operation of other components. All components of the encoding / decoding system 200 can be used in the encoder. The decoder may include a subset of the components of the encoding / decoding system 200. For example, the decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described here.

[0059] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning patterns to subdivide pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into even smaller blocks. These blocks can be called nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is called the node / coding tree depth. In some cases, a coding unit (CU) may include the partitioned blocks. For example, a CU can be a subpart of a CTU, which contains luma blocks, red chroma (Cr) blocks, and blue chroma (Cb) blocks, as well as the corresponding syntax instructions for the CU. Partitioning patterns can include binary trees (BT), triple trees (TT), and quad trees (QT), used to divide nodes into two, three, or four child nodes of different shapes, depending on the partitioning pattern used. The segmented video signal 201 is forwarded to the general decoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control and analysis component 227, and the motion estimation component 221 for compression.

[0060] The general-purpose decoder control component 211 makes decisions related to encoding images of a video sequence into a bitstream based on application constraints. For example, the general-purpose decoder control component 211 manages the optimization of bitrate / bitstream size relative to reconstruction quality. Such decisions can be made based on storage space / bandwidth availability and image resolution requests. The general-purpose decoder control component 211 also manages buffer utilization based on transmission speed to mitigate buffer underloading and overloading issues. To address these issues, the general-purpose decoder control component 211 manages segmentation, prediction, and filtering performed by other components. For example, the general-purpose decoder control component 211 can dynamically increase compression complexity to improve resolution and bandwidth utilization, or decrease compression complexity to reduce resolution and bandwidth utilization. Therefore, the general-purpose decoder control component 211 controls other components of the encoding / decoding system 200 to balance video signal reconstruction quality with bitrate. The general-purpose decoder control component 211 creates control data that controls the operation of other components. Control data is also forwarded to the header format and CABAC component 231 to be encoded into the bitstream, thereby indicating the parameters used for decoding in the decoder.

[0061] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or stripes of the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to provide timing prediction. The encoding / decoding system 200 can perform multiple decoding processes to select an appropriate decoding mode for each video data block, etc.

[0062] Motion estimation component 221 and motion compensation component 219 can be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation component 221 is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, motion vectors can represent the displacement of a decoded object relative to a prediction block. A prediction block is a block found to be highly matched to the block to be decoded in terms of pixel differences. A prediction block can also be referred to as a reference block. Such pixel differences can be determined using the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. HEVC uses several decoded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be divided into multiple CTBs, and then a CTB can be divided into multiple CBs, which are used to be included in a CU. A CU can be encoded as a prediction unit (PU) containing prediction data and / or a transform unit (TU) containing transform residual data of the CU. Motion estimation component 221 uses rate-distortion analysis as part of the rate-distortion optimization process to generate motion vectors, PU, ​​and TU. For example, motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc., for the current block / frame, and can select reference blocks, motion vectors, etc., with optimal rate-distortion characteristics. Optimal rate-distortion characteristics balance the quality of video reconstruction (e.g., the amount of data loss due to compression) with decoding efficiency (e.g., the final encoded size).

[0063] In some examples, the codec system 200 can calculate the values ​​of sub-integer pixel positions of the reference image stored in the decoded image buffer component 223. For example, the video codec system 200 can interpolate the values ​​of quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Therefore, the motion estimation component 221 can perform motion search relative to full-pixel positions and fractional pixel positions, and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates the motion vector of the PU in the inter-frame decoded stripe by comparing the position of the PU with the position of the predicted block in the reference image. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header format and CABAC component 231 for encoding, and outputs the motion to the motion compensation component 219.

[0064] The motion compensation performed by motion compensation component 219 may involve acquiring or generating prediction blocks based on motion vectors determined by motion estimation component 221. Additionally, in some examples, motion estimation component 221 and motion compensation component 219 may be functionally integrated. After receiving the motion vector of the PU for the current video block, motion compensation component 219 can locate the prediction block to which the motion vector points. Then, a residual video block is formed by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being decoded, resulting in a pixel difference. Typically, motion estimation component 221 performs motion estimation relative to the luma component, and motion compensation component 219 uses the motion vectors calculated based on the luma component for both the chroma and luma components. The prediction block and residual block are then forwarded to transform scaling and quantization component 213.

[0065] The segmented video signal 201 is also sent to the intra-frame estimation component 215 and the intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but are shown separately for conceptual purposes. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction relative to blocks in the current frame, instead of the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines an intra-frame prediction mode for encoding the current block. In some examples, the intra-frame estimation component 215 selects an appropriate intra-frame prediction mode from multiple tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then forwarded to the header format and CABAC component 231 for encoding.

[0066] For example, intra-prediction component 215 uses rate-distortion analysis of various tested intra-prediction modes to calculate rate-distortion values ​​and selects the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between a coded block and the original uncoded block encoded to produce the coded block, as well as the bit rate (e.g., number of bits) used to generate the coded block. Intra-prediction component 215 calculates a ratio based on the distortion and rate of various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block. Furthermore, intra-prediction component 215 can be used to decode depth blocks of a depth map using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0067] When implemented on the encoder, intra-prediction component 217 can generate residual blocks from the prediction blocks according to the selected intra-prediction mode determined by intra-estimation component 215, or, when implemented on the decoder, read residual blocks from the bitstream. The residual blocks comprise the value differences between the prediction blocks and the original blocks, represented as a matrix. The residual blocks are then forwarded to transform-scaling and quantization component 213. Intra-estimation component 215 and intra-prediction component 217 can operate on both the luma and chroma components.

[0068] Transform scaling and quantization component 213 is used to further compress the residual block. Transform scaling and quantization component 213 applies a transform to the residual block, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to produce a video block that includes the values ​​of the residual transform coefficients. Wavelet transform, integer transform, subband transform, or other types of transforms can also be used. The transform can convert the residual information from the pixel value domain to the transform domain, such as the frequency domain. Transform scaling and quantization component 213 is also used to scale the residual information of the transform according to frequencies, etc. This scaling involves applying a scaling factor to the residual information to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. Transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, transform scaling and quantization component 213 may then scan a matrix that includes the quantized transform coefficients. The quantization transform coefficients are forwarded to the header format and CABAC component 231 for encoding into the bitstream.

[0069] The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, inverse transform, and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, to be used later as a reference block, which can become the prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can compute the reference block by adding the residual block back to the corresponding prediction block for motion estimation in subsequent blocks / frames. Filters are used to reconstruct the reference block to reduce artifacts generated during scaling, quantization, and transform. Such artifacts can lead to inaccurate predictions (and generate other artifacts) when predicting subsequent blocks.

[0070] The filter control analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, the transform residual block of the scaling and inverse transform component 229 can be merged with the corresponding prediction block of the intra-prediction component 217 and / or motion compensation component 219 to reconstruct the original image block. The filter can then be used to reconstruct the image block. In some examples, the filter can be used instead for the residual block. As... Figure 2 The other components, filter control analysis component 227 and in-loop filter component 225, are highly integrated and could be implemented together, but are shown separately for conceptual purposes. Filters used to reconstruct the reference block are for a specific spatial region and include multiple parameters to adjust how such filters are applied. Filter control analysis component 227 analyzes the reconstructed reference block to determine where such filters should be applied and sets the corresponding parameters. This data is forwarded as filter control data to the header format and CABAC component 231 for encoding. In-loop filter component 225 applies such filters based on the filter control data. Filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be used in the spatial / pixel domain (e.g., for reconstructed pixel blocks) or the frequency domain, depending on the example.

[0071] When used as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for later use in motion estimation, as described above. When used as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and forwards them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0072] The header format and CABAC component 231 receive data from various components of the encoding / decoding system 200 and encode this data into an encoded bitstream for transmission to the decoder. Specifically, the header format and CABAC component 231 generate various headers to encode control data (such as general control data and filter control data). Furthermore, prediction data (including intra-frame prediction) and motion data, as well as residual data in the form of quantization transform coefficient data, are encoded in the bitstream. The final bitstream contains all the information required by the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword map), definitions of the coding context of various blocks, indications of the most likely intra-frame prediction modes, indications of segmentation information, etc. This data can be encoded using entropy decoding techniques. For example, information can be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy decoding techniques. After entropy decoding, the encoded bitstream can be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0073] Figure 3 This is a block diagram of an exemplary video encoder 300. The video encoder 300 can be used to implement the encoding function of the encoding / decoding system 200 and / or implement steps 101, 103, 105, 107, and / or 109 of the operation method 100. The encoder 300 segments the input video signal to generate a segmented video signal 301, wherein the segmented video signal 301 is substantially similar to the segmented video signal 201. The segmented video signal 301 is then compressed by components of the encoder 300 and encoded into a bitstream.

[0074] Specifically, the segmented video signal 301 is forwarded to the intra-frame prediction component 317 for intra-frame prediction. The intra-frame prediction component 317 can be substantially similar to the intra-frame estimation component 215 and the intra-frame prediction component 217. The segmented video signal 301 is also forwarded to the motion compensation component 321 for inter-frame prediction based on the reference block in the decoded image buffer component 323. The motion compensation component 321 can be substantially similar to the motion estimation component 221 and the motion compensation component 219. The predicted blocks and residual blocks of the intra-frame prediction component 317 and the motion compensation component 321 are forwarded to the transform and quantization component 313 for transforming and quantizing the residual blocks. The transform and quantization component 313 can be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual blocks and the corresponding predicted blocks (along with associated control data) are forwarded to the entropy decoding component 331 for encoding into the bitstream. The entropy decoding component 331 can be substantially similar to the header format and CABAC component 231.

[0075] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transformed and quantized component 313 to the inverse transformed and quantized component 329 to reconstruct a reference block for use by the motion compensation component 321. The inverse transformed and quantized component 329 can be substantially similar to the scaling and inverse transformed component 229. According to the example, the in-loop filter in the in-loop filter component 325 is also used for the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. As described with respect to the in-loop filter component 225, the in-loop filter component 325 can include multiple filters. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block by the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.

[0076] Figure 4 This is a block diagram of an exemplary video decoder 400. The video decoder 400 can be used to implement the decoding function of the encoding / decoding system 200 and / or implement steps 111, 113, 115, and / or 117 of the operation method 100. For example, the decoder 400 receives a bitstream from the encoder 300 and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0077] The bitstream is received by entropy decoding component 433. Entropy decoding component 433 is used to implement entropy decoding schemes such as CAVLC, CABAC, SBAC, PIPE decoding, or other entropy decoding techniques. For example, entropy decoding component 433 can use header information to provide context for interpreting other data encoded as codewords in the bitstream. Decoding information includes any information required to decode the video signal, such as general control data, filter control data, segmentation information, motion data, prediction data, and quantization transform coefficients in the residual block. The quantization transform coefficients are forwarded to inverse transform and quantization component 429 to reconstruct the residual block. Inverse transform and quantization component 429 may be similar to inverse transform and quantization component 329.

[0078] The reconstructed residual block and / or predicted block are forwarded to the intra-prediction component 417 to reconstruct image blocks according to the intra-prediction operation. The intra-prediction component 417 can be similar to the intra-estimation component 215 and the intra-prediction component 217. Specifically, the intra-prediction component 417 uses a prediction mode to locate a reference block in the frame and uses the residual block as a result to reconstruct the intra-predicted image block. The reconstructed intra-predicted image block and / or residual block, along with the corresponding inter-frame prediction data, are forwarded to the decoded image buffer component 423 via the intra-loop filter component 425. The decoded image buffer component 423 and the intra-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the intra-loop filter component 225, respectively. The intra-loop filter component 425 filters the reconstructed image block, the residual block, and / or the predicted block, and this information is stored in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter-frame prediction. The motion compensation component 421 can be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses the motion vectors of the reference block to generate a prediction block and uses the residual block as a result to reconstruct an image block. The resulting reconstructed block can also be forwarded to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 continues to store other reconstructed image blocks, which can be reconstructed into frames using segmentation information. Such frames can also be located in a sequence. This sequence is output to the display as a reconstructed output video signal.

[0079] Figure 5 This is a schematic diagram of multiple sub-image video streams 501, 502, and 503 extracted from image video stream 500. For example, according to method 100, each of the sub-image video streams 501-503 and / or image video stream 500 can be encoded by an encoder such as encoding / decoding system 200 and / or encoder 300. Furthermore, the sub-image video streams 501-503 and / or image video stream 500 can be decoded by a decoder such as encoding / decoding system 200 and / or decoder 400.

[0080] Image video stream 500 includes multiple images presented over time. For example... Figure 5 As shown, the image video stream 500 is used for virtual reality (VR) applications. VR operates by decoding the video content of a sphere, which can be displayed as if the user were at the center of the sphere. Each image includes the entire sphere. However, only a portion of the image (called a viewpoint) is shown to the user. For example, the user can use a head-mounted display (HMD) that selects and displays the viewpoint of the sphere based on the user's head movements. This creates the effect of the video depicting a physically existing sphere in virtual space. To achieve this, each image in the video sequence includes the entire video data sphere at the corresponding moment. However, only a small portion of the image (e.g., a single viewpoint) is shown to the user. The rest of the image is discarded in the decoder and not presented. The entire image can be sent so that different viewpoints can be dynamically selected and displayed in response to user head movements.

[0081] In the example shown, the images in image video stream 500 can be subdivided into sub-images based on available viewpoints. Therefore, each image and its corresponding sub-image includes a temporal position (e.g., image order) as part of the temporal presentation. Sub-image video streams 501-503 are created by consistently applying subdivision over a period of time. This consistent subdivision creates sub-image video streams 501-503, where each stream contains a set of sub-images with a predetermined size, shape, and spatial position relative to a corresponding image in image video stream 500. Furthermore, the temporal position of the sub-image sets in sub-image video streams 501-503 changes over presentation time. Therefore, the sub-images in sub-image video streams 501-503 can be aligned in the temporal domain based on their temporal positions. The sub-images in sub-image video streams 501-503 at each temporal position can then be fused in the spatial domain according to predefined spatial positions to reconstruct image video stream 500 for display. Specifically, each of sub-image video streams 501-503 can be encoded into a different sub-bitstream. When these sub-streams are merged together, a single stream is generated that encompasses the entire image set over time. This resulting stream can then be sent to a decoder for decoding and display based on the user's currently selected viewpoint.

[0082] One problem with VR video is that all sub-image video streams 501-503 can be sent to the user in high quality (e.g., high resolution). This allows the decoder to dynamically select the user's current viewpoint and display the corresponding sub-image from sub-image video streams 501-503 in real time. However, the user can only view a single viewpoint, such as from sub-image video stream 501, while sub-image video streams 502-503 are discarded. Therefore, sending sub-image video streams 502-503 in high quality consumes a significant amount of bandwidth without providing a corresponding benefit to the user. To improve decoding efficiency, VR video can be encoded into multiple video streams 500, where each video stream 500 is encoded at a different quality / resolution. In this way, the decoder can send a request for the current sub-image video stream 501. In response, the encoder (or intermediate striper or other content server) can select the higher-quality sub-image video stream 501 from the higher-quality video stream 500 and the lower-quality sub-image video streams 502-503 from the lower-quality video stream 500. The encoder can then fuse these sub-streams into a single, complete encoded stream for transmission to the decoder. In this way, the decoder receives a series of images, with higher quality for the current viewpoint and lower quality for other viewpoints. Furthermore, the highest quality sub-images are typically displayed to the user (when the head is not moving), while lower-quality sub-images are usually discarded, balancing functionality with decoding efficiency.

[0083] When a user switches from watching sub-image video stream 501 to sub-image video stream 502, the decoder requests that the new current sub-image video stream 502 be sent at higher quality. The encoder can then change the fusion mechanism accordingly.

[0084] This invention includes an image / video stream 500 to describe the practical applications of sub-images. It should be noted that sub-images have many applications, and the invention is not limited to VR technology. For example, sub-images can also be used in teleconferencing systems. In this case, each user's video feed is included in a sub-image stream, such as sub-image video streams 501, 502, and / or 503. The system can receive such sub-image video streams 501, 502, and / or 503 and combine them at different locations, at different resolutions, etc., to create a complete image / video stream 500 for transmission back to the user. This allows the teleconferencing system to dynamically change the image / video stream 500 based on changing user input (e.g., by increasing or decreasing the size of sub-image video streams 501, 502, and / or 503) to emphasize the currently speaking user or de-emphasize users who are no longer speaking. Therefore, sub-images have many applications that allow the image / video stream 500 to dynamically change according to changes in user behavior during runtime. This function can be achieved by extracting sub-image video streams 501, 502 and / or 503 from image video stream 500, and / or combining sub-image video streams 501, 502 and / or 503 into image video stream 500.

[0085] Figure 6 This is a schematic diagram of an exemplary bitstream 600 divided into sub-bitstreams 601. Bitstream 600 may contain image / video streams, such as image / video stream 500, and sub-bitstream 601 may contain sub-image / video streams, such as sub-image / video streams 501, 502, and / or 503. For example, bitstream 600 and sub-bitstream 601 may be generated by encoding / decoding system 200 and / or encoder 300 for decoding by encoding / decoding system 200 and / or decoder 400. As another example, in step 109 of method 100, bitstream 600 and sub-bitstream 601 may be generated by encoder for use by decoder (step 111).

[0086] Bitstream 600 includes a sequence parameter set (SPS) 610, multiple picture parameter sets (PPS) 611, multiple stripe headers 615, and image data 620. SPS 610 contains sequence data shared by all images in the video sequence contained in bitstream 600. This data may include image size, bit depth, decoding tool parameters, bit rate limits, etc. PPS 611 contains parameters applied to the entire image. Therefore, each image in the video sequence can reference PPS 611. It should be noted that although each image references PPS 611, in some examples, a single PPS 611 may contain data from multiple images. For example, multiple similar images can be encoded according to similar parameters. In this case, a single PPS 611 can contain data from these similar images. PPS 611 may represent quantization parameters, offsets, decoding tools that can be used for the stripes in the corresponding image, etc. Stripe headers 615 contain parameters specific to each strip in the image. Therefore, each strip in the video sequence can have a stripe header 615. The slice header 615 may contain slice type information, picture order count (POC), a list of reference images, prediction weights, tile entry points, deblocking filter parameters, etc. It should be noted that in some contexts, the slice header 615 may also be called the tile group header. It should also be noted that in some examples, the bitstream 600 may also include an image header, which is a syntax structure containing parameters applied to all slices in a single image. Therefore, in some contexts, the image header and the slice header 615 can be used interchangeably. For example, the slice header 615 and the image header may use some of the same parameters, depending on whether these parameters are shared by all slices in the image.

[0087] Image data 620 contains video data encoded according to inter-frame prediction, intra-frame prediction, and / or inter-layer prediction, as well as corresponding transform and quantization residual data. For example, a video sequence includes multiple images 621. Images 621 are arrays of luminance samples and / or chrominance samples that create frames or frame fields. A frame is a complete image in the video sequence that is expected to be displayed fully or partially to the user at a corresponding moment. Images 621 contain one or more stripes. A stripe can be defined as an integer number of complete blocks in images 621 or an integer number of consecutive complete CTU lines in images 621 (e.g., within blocks), which are contained in a single NAL unit. Stripes are further divided into CTUs and / or CTBs. A CTU is a set of samples of a predefined size that can be segmented by a coding tree. A CTB is a subset of CTUs and contains the luminance or chrominance components of the CTUs. CTUs / CTBs are further divided into coding blocks according to the coding tree. The coding blocks can then be encoded / decoded according to the prediction mechanism.

[0088] Image 621 can be divided into multiple sub-images 623 and 624. Sub-images 623 and / or 624 are rectangular regions composed of one or more stripes in image 621. Therefore, each strip and its subdivisions can be assigned to sub-images 623 and / or 624. This allows different regions of image 621 to be processed differently from a decoding perspective, depending on which sub-images 623 and / or 624 are included in those regions.

[0089] According to the sub-stream extraction process 605, sub-stream 601 can be extracted from bitstream 600. Sub-stream extraction process 605 is a specified mechanism that removes NAL units from the bitstream that are not part of the target set, thereby producing an output sub-stream that includes the NAL units included in the target set. NAL units contain stripes. Therefore, sub-stream extraction process 605 retains the target set of stripes and removes other stripes. The target set can be selected based on sub-image boundaries. In the example shown, the stripe included in sub-image 623 is included in the target set, and the stripe included in sub-image 624 is not included in the target set. Therefore, sub-stream extraction process 605 creates sub-stream 601, which is substantially similar to bitstream 600 but includes sub-image 623 while excluding sub-image 624. Sub-stream extraction process 605 can be performed by an encoder and / or an associated striper used to dynamically change bitstream 600 based on user behavior / requests.

[0090] Therefore, sub-stream 601 is the bitstream extracted by applying sub-stream extraction process 605 to input bitstream 600. Input bitstream 600 contains a set of sub-images. However, the extracted bitstream (e.g., sub-stream 601) contains only a subset of the sub-images of input bitstream 600 used in sub-stream extraction process 605. In the example shown, the set of sub-images included in input bitstream 600 includes sub-images 623 and 624, while the subset of sub-images in sub-stream 601 includes sub-image 623 but not sub-image 624. It should be noted that any number of sub-images 623-624 can be used. For example, bitstream 600 may include N sub-images 623-624, and sub-stream 601 may contain N-1 or fewer sub-images 623, where N is any integer value.

[0091] In some cases, the sub-stream extraction process 605 may produce decoding errors. For example, sub-images 623-624 may be associated with sub-image information (e.g., sub-image IDs). A sub-image ID uniquely identifies the corresponding sub-image, such as sub-image 623 or 624. Therefore, the sub-image ID can be used to indicate the position of sub-images 623-624 relative to image 621 and / or to modify the sub-image level decoding process. In some cases, sub-image information can be inferred from the position of sub-images 623-624. Therefore, the bitstream 600 can omit this sub-image information associated with sub-images 623 and 624 to reduce the amount of data in the bitstream 600 and improve decoding efficiency. However, when sub-image 623 or sub-image 624 is absent, the decoder may be unable to infer this sub-image information. Therefore, a simple sub-stream extraction process 605 can be applied to a consistent bitstream 600 and create an inconsistent sub-stream 601. When bitstream 600 / sub-bitstream 601 conforms to a standard (e.g., VVC), bitstream 600 / sub-bitstream 601 are consistent and can therefore be correctly decoded by any decoder that also conforms to the standard. Therefore, a simple sub-bitstream extraction process 605 can convert the decodeable bitstream 600 into the undecodeable sub-bitstream 601.

[0092] To address this issue, the present invention includes an improved substream extraction process 605. Specifically, the substream extraction process 605 encodes the subimage IDs of subimages 623 in substream 601, even when these subimage IDs are omitted from substream 600. For example, the subimage IDs may be included in the SPS subimage identifier (sps_subpic_id[i]) syntax structure 635. The sps_subpic_id[i] syntax structure 635 is included in SPS 610 and includes i subimage IDs, where i is the number of subimages 623 included in substream 601. Furthermore, the substream extraction process 605 may also encode the bit length of the syntax element containing one or more subimage IDs (e.g., the sps_subpic_id[i] syntax structure 635) into the extracted bitstream. For example, this length may be included in the SPS subimage ID length minus one (sps_subpic_id_len_minus1) syntax structure 633. The sps_subpic_id_len_minus1 syntax structure 633 can contain the bit length of the sps_subpic_id[i] syntax structure 635 minus one. The minus-one decoding method encodes the value as one less than the actual value to save bits. The decoder can deduce the actual value by adding one. Therefore, the sps_subpic_id_len_minus1 syntax structure 633 can also be referred to as sps_subpic_id_len_minus1+1. Thus, the decoder can use the sps_subpic_id_len_minus1 syntax structure 633 to determine the number of bits associated with the sps_subpic_id[i] syntax structure 635, and therefore can use the sps_subpic_id_len_minus1 syntax structure 633 to interpret the sps_subpic_id[i] syntax structure 635. The decoder can then decode the sub-image 623 based on the sps_subpic_id_len_minus1 syntax structure 633 and the sps_subpic_id[i] syntax structure 635.

[0093] Furthermore, the sub-stream extraction process 605 can encode / set a flag in the sub-stream 601 to indicate the presence of sub-image information related to the sub-image 623 in the sub-stream 601. In a specific example, this flag can be encoded as a sub-image information presence flag (subpic_info_present_flag) 631. Therefore, subpic_info_present_flag 631 can be set to indicate the presence of sub-image information related to a subset of the sub-images in the extracted bitstream (sub-stream 601), such as the sps_subpic_id_len_minus1 syntax structure 633 and the sps_subpic_id[i] syntax structure 635. Additionally, the decoder can read subpic_info_present_flag 631 to determine the presence of sub-image information related to a subset of the sub-images in the extracted bitstream (sub-stream 601), such as the sps_subpic_id_len_minus1 syntax structure 633 and the sps_subpic_id[i] syntax structure 635. In a specific example, when the extracted bitstream (sub-bitstream 601) is extracted from the input bitstream 600 through the sub-bitstream extraction process 605, the encoder / striper can request that a flag be set to 1 to indicate that sub-image information exists in the coded layer video sequence (CLVS), and that each image 621 of the CLVS contains multiple sub-images 623 and 624. A CLVS is a sequence of coded video data containing one or more layers of images. A layer is a set of NAL units, each with a specific layer ID value. Images 621 may or may not be organized into multiple layers, where all images 621 of a corresponding layer have similar characteristics, such as size, resolution, signal-to-noise ratio (SNR), etc.

[0094] The above information is described in detail below. HEVC can employ regular stripes, correlated stripes, block partitioning, and wavefront parallel processing (WPP) as segmentation schemes. These segmentation schemes can be used for maximum transfer unit (MTU) size matching, parallel processing, and reducing end-to-end latency. Each regular stripe can be encapsulated in a separate NAL unit. At stripe boundaries, entropy decoding dependencies and intra-frame prediction (including intra-sample prediction, motion information prediction, and decoding mode prediction) can be disabled. Therefore, regular stripes can be reconstructed independently of other regular stripes in the same image. However, due to loop filtering operations, stripes may still have some interdependencies.

[0095] Parallelization based on regular stripes may not require significant inter-processor or inter-core communication. One exception is when decoding a predictive decoded image, where inter-processor and / or inter-core data sharing can be beneficial for motion compensation. This process may involve more processing resources than inter-processor or inter-core data sharing due to inter-frame prediction. However, for the same reason, using regular stripes can result in significant decoding overhead due to the bit cost of the stripe header and the lack of prediction across stripe boundaries. Furthermore, due to the intra-frame independence of regular stripes and the fact that each regular stripe is encapsulated in a separate NAL unit, regular stripes are also used as a mechanism for bitstream segmentation to match MTU size requirements. In many cases, the goal of achieving parallelization and MTU size matching presents conflicting requirements for the stripe layout within the image.

[0096] Correlated slices feature a shortened slice header and support bitstream segmentation at tree block boundaries without interrupting any intra-frame prediction. Correlated slices also provide the ability to divide a regular slice into multiple NAL units. This allows for the transmission of a portion of the regular slice before the entire regular slice is encoded, thus reducing end-to-end latency.

[0097] In WPP, the image is segmented into single-line CTBs. Entropy decoding and prediction can utilize data from CTBs in other segmented parts. Parallel processing is supported by decoding CTB lines in parallel. As an example, the decoding of a CTB line can be delayed by one or two CTBs to ensure that data related to the CTBs above and to the right of the target CTB is available before decoding the target CTB. This staggered start results in wavefront parallel processing. This processing can be parallelized using as many processors / cores as the number of CTB lines contained within the image. Since intra-frame prediction can be performed between adjacent tree block lines within the image, it is crucial to use inter-processor / inter-core communication for intra-frame prediction. WPP segmentation does not generate additional NAL units. Therefore, WPP cannot be used for MTU size matching. However, regular striping can be combined with WPP to achieve MTU size matching as needed, which requires some decoding overhead.

[0098] A chunk is a segmented portion of an image created by horizontal and vertical boundaries that define the chunk's columns and rows. The scan order of the CTB can be confined within a chunk, following the order of the chunk's CTB raster scans. Therefore, a chunk can be fully decoded before the top-left CTB of the next chunk is decoded in the order of the chunk raster scans of the image. Similar to regular stripes, chunks eliminate intra-prediction dependencies as well as entropy decoding dependencies. However, chunks may not be included in a single NAL unit. Therefore, chunks cannot be used for MTU size matching. Each chunk can be processed by a single processor / core. Inter-processor / inter-core communication for intra-prediction between processing units used to decode adjacent chunks can be limited to transmitting a shared stripe header when the stripe includes multiple chunks and reconstructed samples and metadata related to loop filtering are shared. When a stripe includes multiple chunks or WPP segments, the entry point byte offset for each chunk or WPP segment can be indicated in the stripe header, in addition to the first chunk or WPP segment in the stripe.

[0099] For simplicity, HEVC imposes certain restrictions on the application of the four different image segmentation schemes. Encoded video sequences cannot simultaneously include both block segmentation and wavefront parallel processing for most profiles specified in HEVC. Furthermore, each strip and / or block must satisfy one or both of the following conditions: All decoded tree blocks in a strip are contained within the same block. Additionally, all decoded tree blocks in a block are contained within the same strip. Furthermore, the wavefront segment contains exactly one CTB line. When using WPP, stripes starting from one CTB line should end at the same CTB line.

[0100] In VVC, a tile is a segmented portion of an image created by horizontal and vertical boundaries, which define the tile's columns and rows. VVC can support further horizontal subdivision of tiles to form bricks. Tiles that are not further subdivided can also be considered bricks. The scan order of the CTB is changed to be confined to within the bricks (e.g., according to the order of the brick's CTB raster scan). The current brick can be fully decoded before decoding the top-left CTB of the next brick in the order of the current brick raster scan of the image.

[0101] In VVC, a strip can include one or more bricks. Each strip is encapsulated in a separate NAL unit. Entropy decoding dependency and intra-frame prediction (including intra-sample prediction, motion information prediction, and decoding mode prediction) can be disabled at strip boundaries. Therefore, regular strips can be reconstructed independently of other regular strips in the same image. VVC includes rectangular strips and raster scan strips. A rectangular strip can include one or more bricks occupying a rectangular region in the image. A raster scan strip can include one or more bricks arranged in the raster scan order of the bricks in the image.

[0102] VVC-based WPP is similar to HEVC WPP, but HEVC WPP has two CTU delays, while VVC WPP has one. For HEVC WPP, after the first two CTUs in the previous CTU row are decoded, a new decoding thread can begin decoding the first CTU in the allocated CTU row. For VVC WPP, after the first CTU in the previous CTU row is decoded, a new decoding thread can begin decoding the first CTU in the allocated CTU row.

[0103] The following are exemplary instructions for blocks, bricks, and stripes in PPS.

[0104]

[0105]

[0106]

[0107] The above system has some issues. For example, when the bitstream is first encoded, the slices in the image within the bitstream can be segmented into rectangular slices. In this case, the slice ID can be omitted in the PPS. In this situation, the value of `signalled_slice_id_flag` in the bitstream's PPS can be set to 0. However, when one or more rectangular slices are extracted from the bitstream to form another bitstream, the slice ID should be present in the PPS of the bitstream generated from this extraction process.

[0108] This invention typically describes the delivery of indicator strip IDs to aid in the bitstream extraction process. These techniques are based on VVC, but can also be applied to other video codec specifications.

[0109] An exemplary mechanism for solving the above problem is as follows. A method is disclosed for extracting one or more slices from an image of a bitstream represented as bitstream A and generating a new bitstream B through an extraction process. Bitstream A includes at least one image, which includes multiple slices. The method includes parsing a parameter set from bitstream A and rewriting the parameter set into bitstream B. In the rewritten parameter set, the value of `signalled_slice_id_flag` is set to 1. When the `signalled_slice_id_length_minus1` syntax element exists in the parameter set of bitstream A, the value of `signalled_slice_id_flag` is copied to the rewritten parameter set. When the `signalled_slice_id_length_minus1` syntax element does not exist in the parameter set of bitstream A, the value of `signalled_slice_id_flag` is set in the rewritten parameter set. For example, `signalled_slice_id_flag` can be set to `Ceil(Log2(num_slices_in_pic_minus1+1))-1`, where `num_slices_in_pic_minus1` equals the number of slices in the image of bitstream A minus 1. One or more slices are extracted from bitstream A. Then, the extracted bitstream B is generated.

[0110] The exemplary PPS semantics are as follows. A `signalled_slice_id_flag` set to 1 indicates that the slice ID for each slice is indicated. A `signalled_slice_id_flag` set to 0 indicates that no slice ID is indicated. When `rect_slice_flag` equals 0, the value of `signalled_slice_id_flag` can be inferred to be 0. For a bitstream extracted via a sub-stream extraction process, which contains a subset of the slices initially included in the image, the value of `signalled_slice_id_flag` should be set to 1 for PPS. `signaled_slice_id_length_minus1+1` can represent the number of bits used to represent the syntax element `slice_id[i]` (if present) and the syntax element `slice_address` in the slice header. The value of `signaled_slice_id_length_minus1` can range from 0 to 15 (inclusive). When it does not exist, the value of signed_slice_id_length_minus1 can be inferred to be equal to Ceil(Log2(num_slices_in_pic_minus1+1))-1. For bitstreams extracted through the sub-bitstream extraction process, which contains a subset of the stripes originally included in the image, the value of signed_slice_id_length_minus1 should remain unchanged for PPS.

[0111] Figure 7This is a schematic diagram of an exemplary video decoding device 700. The video decoding device 700 is suitable for implementing the disclosed examples / embodiments described herein. The video decoding device 700 includes a downlink port 720, an uplink port 750, and / or a transceiver unit (Tx / Rx) 710. The transceiver unit (Tx / Rx) 710 includes a transmitter and / or receiver for transmitting data upstream and / or downstream over a network. The video decoding device 700 also includes: a processor 730 including a logic unit and / or a central processing unit (CPU) for processing data; and a memory 732 for storing data. The video decoding device 700 may also include electrical components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to the uplink port 750 and / or the downlink port 720 for data transmission over an electrical, optical, or wireless communication network. The video decoding device 700 may also include input and / or output (I / O) devices 760 for data communication with a user. I / O devices 760 may include output devices, such as a display for showing video data, a speaker for outputting audio data, etc. I / O devices 760 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with these output devices.

[0112] Processor 730 is implemented in both hardware and software. Processor 730 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). Processor 730 communicates with downlink port 720, Tx / Rx 710, uplink port 750, and memory 732. Processor 730 includes a decoding module 714. Decoding module 714 implements the disclosed embodiments described herein, such as methods 100, 800, and / or 900, and may use bitstream 600 and / or sub-bitstream 601, including image video stream 500 and / or sub-image video streams 501-503. Decoding module 714 may also implement any other methods / mechanisms described herein. Furthermore, decoding module 714 may implement codec system 200, encoder 300, and / or decoder 400. For example, the decoding module 714 can be used to extract sub-streams from the bitstream, include sub-image information in the sub-stream during the extraction process, and / or include flags in the sub-stream to indicate that sub-image information is included in the sub-stream. Therefore, the decoding module 714 enables the video decoding device 700 to provide additional functionality and / or improve decoding efficiency when decoding video data. Thus, the decoding module 714 improves the functionality of the video decoding device 700 and solves problems specific to the field of video coding. Furthermore, the decoding module 714 affects the transitions of the video decoding device 700 to different states. Alternatively, the decoding module 714 can be implemented as instructions (e.g., a computer program product stored in a non-transitory medium) stored in memory 732 and executed by processor 730.

[0113] Memory 732 includes one or more memory types, such as disks, tape drives, solid-state drives, read-only memory (ROM), random access memory (RAM), flash memory, ternary content-addressable memory (TCAM), static random-access memory (SRAM), etc. Memory 732 can be used as an overflow data storage device to store programs when a program is selected for execution, and to store instructions and data read during program execution.

[0114] Figure 8This is a flowchart of an exemplary method 800 for encoding a video sequence into a bitstream (e.g., bitstream 600) and extracting sub-bitstreams (e.g., sub-bitstream 601) while reducing ID errors. When method 100 is performed to encode image video stream 500 and / or sub-image video streams 501-503, method 800 can be performed by an encoder (e.g., codec system 200, encoder 300, and / or video decoding device 700).

[0115] Method 800 may begin with: an encoder receiving a video sequence comprising multiple images and determining, based on user input, to encode the video sequence into a bitstream. In step 801, the encoder encodes an input bitstream (e.g., image video stream 500 and / or bitstream 600) containing a set of sub-images. For example, the bitstream may contain VR video data and / or teleconference video data. The set of sub-images may contain multiple sub-images. Furthermore, sub-images may be associated with sub-image IDs.

[0116] In step 803, the encoder and / or associated striper perform a substream extraction process on the input bitstream to create extracted bitstreams, such as sub-image video streams 501-503 and / or substream 601. The extracted bitstream contains only a subset of the sub-images of the input bitstream. Specifically, the extracted bitstream contains only the sub-images included in the sub-image set of the input bitstream. Furthermore, the extracted bitstream excludes one or more sub-images from the sub-image set of the input bitstream. Thus, the input bitstream may contain the CLVS of images, while the extracted bitstream contains the CLVS of sub-images of images.

[0117] In step 805, the encoder encodes one or more sub-image IDs of a subset of sub-images in the extracted bitstream into the extracted bitstream. For example, these sub-image IDs can be excluded from the input bitstream. Therefore, the encoder can encode these sub-image IDs into the extracted bitstream to support decoding of the sub-images contained in the extracted bitstream. For example, the sub-image IDs can be included in / encoded into the sps_subpic_id[i] syntax structure in the extracted bitstream.

[0118] In step 807, the encoder encodes the bit length of the syntax element containing one or more sub-image IDs into the extracted bitstream. For example, the length of the sub-image IDs can be excluded from the input bitstream. Therefore, the encoder can encode the length of the sub-image IDs into the extracted bitstream to support decoding of the sub-images contained in the extracted bitstream. For example, the length can be included in / encoded into the sps_subpic_id_len_minus1+1 syntax structure in the extracted bitstream.

[0119] In step 809, the encoder may set a flag in the extracted bitstream to indicate the presence of sub-image information associated with a subset of sub-images. This flag may indicate to the decoder the presence of a sub-image ID and / or the length of that sub-image ID in the extracted bitstream. For example, this flag could be `subpic_info_present_flag`. In a specific example, when the extracted bitstream is extracted from the input bitstream via a sub-bitstream extraction process, the flag needs to be set to 1 to indicate the presence of sub-image information in the CLVS (e.g., contained in the input bitstream and / or the extracted bitstream), and that each image in the CLVS contains multiple sub-images. In some examples, the flag, sub-image ID, and length are encoded into the SPS in the extracted bitstream.

[0120] In step 811, the encoder stores the bitstream for communication with the decoder. In some examples, the bitstream can then be sent to the decoder. For example, the bitstream can be sent to the decoder upon request from the decoder, based on a user request, etc.

[0121] Figure 9 This is a flowchart of an exemplary method 900 for decoding a video sequence from a sub-bitstream (e.g., sub-bitstream 601) extracted from a bitstream (e.g., bitstream 600). When method 100 is performed to decode image video stream 500 and / or sub-image video streams 501-503, method 900 can be executed by a decoder (e.g., codec system 200, decoder 400, and / or video decoding device 700).

[0122] For example, after method 800 ends, method 900 can begin when the decoder starts receiving sub-bitstreams extracted from the bitstream. In step 901, the decoder receives the extracted bitstream. The extracted bitstream is the bitstream extracted from the input bitstream containing a set of sub-images through a sub-bitstream extraction process. The extracted bitstream contains only a subset of the sub-images of the input bitstream used for the sub-bitstream extraction process. Specifically, the extracted bitstream contains only the sub-images included in the sub-image set in the input bitstream. Furthermore, the extracted bitstream excludes one or more sub-images from the sub-image set in the input bitstream. Thus, the input bitstream may contain CLVS of images, while the extracted bitstream may contain CLVS of sub-images of images. The received extracted bitstream can also be referred to as a sub-bitstream. For example, the extracted bitstream may contain sub-images including VR video data and / or teleconference video data.

[0123] In step 903, the decoder determines a flag in the extracted bitstream that is set to indicate the presence of sub-image information associated with a subset of sub-images in the extracted bitstream. This flag may indicate the presence of a sub-image ID and / or the length of that sub-image ID in the extracted bitstream. For example, the flag could be `subpic_info_present_flag`. In a specific example, when the extracted bitstream is extracted from the input bitstream via a sub-bitstream extraction process, the flag needs to be set to 1 to indicate that sub-image information exists in the CLVS (e.g., contained in the input bitstream and / or the extracted bitstream), and that each image in the CLVS contains multiple sub-images.

[0124] In step 905, the decoder obtains the bit length of a syntax element containing one or more subpic IDs. For example, the length of the subpic IDs can be excluded from the input bitstream, but the length of the subpic IDs is included in the extracted bitstream. For example, the length can be included in / encoded into the sps_subpic_id_len_minus1+1 syntax structure in the received extracted bitstream.

[0125] In step 907, the decoder obtains one or more sub-image IDs of a subset of sub-images based on a flag and / or a length. For example, the decoder can use a flag to determine the presence of a sub-image ID. The decoder can then use the length to determine the boundaries of the sub-image ID data in the bitstream. For example, a sub-image ID can be excluded from the input bitstream, but it is included in the extracted bitstream. For example, the sub-image ID can be included in / encoded into an `sps_subpic_id[i]` syntax structure in the extracted bitstream. In some examples, the flag, sub-image ID, and length are obtained from the SPS in the extracted bitstream.

[0126] In step 909, the decoder can decode a subset of sub-images in the extracted bitstream based on the sub-image ID obtained in step 907. Then, in step 911, the decoder can forward the subset of sub-images for display as part of the decoded video sequence.

[0127] Figure 10This is a schematic diagram of an exemplary system 1000 for decoding a video sequence of images in a bitstream (e.g., bitstream 600), extracting sub-bitstreams (e.g., sub-bitstream 601), and reducing ID errors. Therefore, system 1000 can be used to decode image video stream 500 and / or sub-image video streams 501-503. System 1000 can be implemented using encoders and decoders (e.g., encoding / decoding system 200, encoder 300, decoder 400, and / or video decoding device 700). Furthermore, system 1000 can be used to implement methods 100, 800, and / or 900.

[0128] System 1000 includes a video encoder 1002. The video encoder 1002 includes a first encoding module 1001 for encoding an input bitstream containing a subset of sub-images. The video encoder 1002 also includes a bitstream extraction module 1004 for performing a sub-bitstream extraction process on the input bitstream to create an extracted bitstream containing only a subset of sub-images from the input bitstream. The video encoder 1002 also includes a second encoding module 1003 for encoding one or more sub-image IDs of the subset of sub-images in the extracted bitstream into the extracted bitstream. The video encoder 1002 also includes a setting module 1005 for setting flags in the extracted bitstream to indicate the presence of sub-image information related to the subset of sub-images in the extracted bitstream. The video encoder 1002 also includes a storage module 1007 for storing the bitstream for communication with a decoder. The video encoder 1002 also includes a sending module 1009 for sending the bitstream to a video decoder 1010. The video encoder 1002 can also be used to perform any step in method 800.

[0129] System 1000 also includes a video decoder 1010. The video decoder 1010 includes a receiving module 1011 for receiving an extracted bitstream, which is extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process. The extracted bitstream contains only a subset of the sub-images of the input bitstream used in the sub-bitstream extraction process. The video decoder 1010 also includes a determining module 1013 for determining a flag in the extracted bitstream, the flag being set to indicate the presence of sub-image information associated with a subset of sub-images in the extracted bitstream. The video decoder 1010 also includes an acquiring module 1015 for acquiring one or more sub-image IDs of the subset of sub-images based on the flag. The video decoder 1010 also includes a decoding module 1017 for decoding the subset of sub-images based on the sub-image IDs. The video decoder 1010 also includes a forwarding module 1019 for forwarding the subset of sub-images for display as part of a decoded video sequence. The video decoder 1010 can also be used to perform any step in method 900.

[0130] When there are no intermediate components between the first and second components other than a line, trace, or other medium, the first and second components are directly coupled. When there are intermediate components between the first and second components other than a line, trace, or other medium, the first and second components are indirectly coupled. The term "coupling" and its synonyms include direct coupling and indirect coupling. Unless otherwise stated, the term "about" means a range including ±10% of the following quantity.

[0131] It should also be understood that the steps of the exemplary methods described herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood as merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and some steps may be omitted or combined.

[0132] While several embodiments have been provided in this invention, it will be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit and scope of the invention. These examples are intended to be illustrative rather than restrictive and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0133] Furthermore, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the invention. Those skilled in the art can identify other examples of changes, substitutions, and modifications, and make such changes, substitutions, and modifications without departing from the spirit and scope of the invention.

Claims

1. A video decoding device, characterized in that, Including processor and memory: The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to perform: Receive bitstream; The bitstream includes: multiple sub-images obtained by dividing an image, and multiple slices obtained by dividing an image; a sequence parameter set SPS, the SPS including a first flag, the first flag being used to indicate whether there is sub-image information in the encoded video sequence CLVS, the first flag also being used to indicate that each image in the CLVS contains multiple sub-images; and an image parameter set PPS, the PPS including a second flag, the second flag being used to indicate whether relevant information about the slice layout is indicated in the PPS; The bitstream is parsed to obtain the first flag and the second flag; The decoded image is obtained by decoding the bitstream based on the first flag and the second flag.

2. The video decoding device according to claim 1, characterized in that, The processor is further configured to retrieve a target bitstream from at least one storage medium and transmit the target bitstream to a destination device via a communication interface.

3. The video decoding device according to claim 1 or 2, characterized in that, The bitstream also includes one or more sub-image identifiers (IDs) of a subset of the sub-images.

4. The video decoding device according to claim 3, characterized in that, The subpic ID is contained in the SPS subpic ID sps_subpic_id[i] syntax element.

5. The video decoding device according to claim 4, characterized in that, The bitstream also includes the syntax element sps_subpic_id_len_minus1, which represents the bit length of sps_subpic_id[i] minus one.

6. The video decoding device according to claim 1 or 2, characterized in that, When the bitstream is a bitstream extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, and the extracted bitstream contains only a subset of the sub-images of the input bitstream, the first flag is set to 1 to indicate that the sub-image information exists in the CLVS, and each image of the CLVS contains multiple sub-images.

7. The video decoding device according to claim 1 or 2, characterized in that, When the second flag is equal to 1, it indicates that the relevant information of the slice layout is indicated in the PPS; when the second flag is equal to 0, it indicates that the relevant information of the slice layout is not indicated in the PPS.

8. A method for storing a bitstream, characterized in that, include: The bitstream is obtained by performing an encoding method, wherein the encoding method includes: The sequence parameter set SPS and the image parameter set PPS are encoded into the bitstream. The SPS includes a first flag, which is used to indicate whether there is sub-image information in the encoded video sequence CLVS. The first flag is also used to indicate that each image in the CLVS contains multiple sub-images. The image is divided into multiple slices, and the multiple slices are encoded into the bitstream; the PPS includes a second flag, which is used to indicate whether the relevant information of the slice layout is indicated in the PPS; The bitstream is stored in one or more storage media.

9. The method for storing a bitstream according to claim 8, characterized in that, Also includes: The target bitstream is retrieved from the one or more storage media and transmitted to the destination device via a communication interface.

10. The method for storing a bitstream according to claim 8 or 9, characterized in that, The bitstream includes multiple sub-image video streams of different resolutions, and the method further includes: Select the target sub-image video stream based on storage space and / or bandwidth availability; The target sub-image video stream is transmitted to the destination device via a communication interface.

11. The method for storing a bitstream according to claim 8 or 9, characterized in that, The bitstream includes multiple sub-image video streams, and the method further includes: In response to a request from the target device, a first sub-image video stream and a second sub-image video stream are selected from the one or more storage media, and the first sub-image video stream and the second sub-image video stream are fused to obtain a fused bitstream; The fused bitstream is transmitted to the destination device.

12. The method for storing a bitstream according to claim 8 or 9, characterized in that, The bitstream also includes one or more sub-image identifiers IDs of a subset of the sub-images, the sub-image IDs being contained in the SPS sub-image identifier sps_subpic_id[i] syntax element.

13. The method for storing a bitstream according to claim 8 or 9, characterized in that, The bitstream also includes the syntax element sps_subpic_id_len_minus1, which represents the bit length of sps_subpic_id[i] minus one.

14. The method for storing a bitstream according to claim 8 or 9, characterized in that, When the bitstream is a bitstream extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, and the extracted bitstream contains only a subset of the sub-images of the input bitstream, the first flag is set to 1 to indicate that the sub-image information exists in the CLVS, and each image of the CLVS contains multiple sub-images.

15. The method for storing a bitstream according to claim 8 or 9, characterized in that, When the second flag is equal to 1, it indicates that the relevant information of the slice layout is indicated in the PPS; when the second flag is equal to 0, it indicates that the relevant information of the slice layout is not indicated in the PPS.

16. A video decoding device, characterized in that, Including processor and memory: The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions to perform: The sequence parameter set SPS and the image parameter set PPS are encoded into the bitstream. The SPS includes a first flag, which is used to indicate whether there is sub-image information in the encoded video sequence CLVS. The first flag is also used to indicate that each image in the CLVS contains multiple sub-images. The image is divided into multiple slices, and the multiple slices are encoded into the bitstream; the PPS includes a second flag, which is used to indicate whether the relevant information of the slice layout is indicated in the PPS.

17. The video decoding device according to claim 16, characterized in that, The bitstream includes multiple sub-image video streams. The processor is further configured to, in response to a request from the decoder, select a first sub-image video stream and a second sub-image video stream from at least one storage medium, fuse the first sub-image video stream and the second sub-image video stream to obtain a fused bitstream, and transmit the fused bitstream to the decoder.

18. The video decoding device according to claim 16 or 17, characterized in that, The processor is also configured to select a target sub-image video stream based on storage space and / or bandwidth availability; The target sub-image video stream is transmitted to the decoder via a communication interface.

19. The video decoding device according to claim 16 or 17, characterized in that, When the bitstream is a bitstream extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, and the extracted bitstream contains only a subset of the sub-images of the input bitstream, the first flag is set to 1 to indicate that the sub-image information exists in the CLVS, and each image of the CLVS contains multiple sub-images.

20. The video decoding device according to claim 16 or 17, characterized in that, When the second flag is equal to 1, it indicates that the relevant information of the slice layout is indicated in the PPS; when the second flag is equal to 0, it indicates that the relevant information of the slice layout is not indicated in the PPS.

21. The video decoding device according to claim 16 or 17, characterized in that, The bitstream also includes one or more sub-image identifiers IDs of a subset of the sub-images, the sub-image IDs being contained in the SPS sub-image identifier sps_subpic_id[i] syntax element.

22. A method for distributing a bitstream, characterized in that, include: The bitstream is obtained by performing an encoding method, wherein the encoding method includes: The sequence parameter set SPS and the image parameter set PPS are encoded into the bitstream. The SPS includes a first flag, which is used to indicate whether there is sub-image information in the encoded video sequence CLVS. The first flag is also used to indicate that each image in the CLVS contains multiple sub-images. The image is divided into multiple slices, and the multiple slices are encoded into the bitstream; the PPS includes a second flag, which is used to indicate whether the relevant information of the slice layout is indicated in the PPS; Receive the first request; In response to the first request, a target bitstream is selected from at least one storage medium and sent to the destination device.

23. The method for distributing a bitstream according to claim 22, characterized in that, The bitstream includes multiple sub-image video streams, and the method further includes: In response to the first request, a first sub-image video stream and a second sub-image video stream are selected from the at least one storage medium, and the first sub-image video stream and the second sub-image video stream are fused to obtain a fused bitstream; The fused bitstream is transmitted to the destination device.

24. The method for distributing a bitstream according to claim 22 or 23, characterized in that, The bitstream includes multiple sub-image video streams, and the method further includes: Select the target sub-image video stream based on storage space and / or bandwidth availability; The target sub-image video stream is transmitted to the destination device via a communication interface.

25. The method for distributing a bitstream according to claim 22 or 23, characterized in that, When the bitstream is a bitstream extracted from an input bitstream containing a set of sub-images through a sub-bitstream extraction process, and the extracted bitstream contains only a subset of the sub-images of the input bitstream, the first flag is set to 1 to indicate that the sub-image information exists in the CLVS, and each image of the CLVS contains multiple sub-images.

26. The method for distributing a bitstream according to claim 22 or 23, characterized in that, When the second flag is equal to 1, it indicates that the relevant information of the slice layout is indicated in the PPS; when the second flag is equal to 0, it indicates that the relevant information of the slice layout is not indicated in the PPS.

Images