Explicit address indication in video decoding
By using stripe identifiers for addressing in video decoding, the computational burden problem of index addressing scheme in virtual reality applications is solved, and the subcode stream extraction and mapping without rewriting stripe headers is realized, reducing resource consumption .
Patent Information
- Application Number
- CN202311864367.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-08-06
- Filing Date
- 2019-12-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2039-12-31
AI Technical Summary
In video decoding, especially in virtual reality applications, index-based addressing schemes may stop operating normally during streaming, because the upper left index of the sub-image received by the decoder is usually not 0, resulting in dynamic rewrites required The stripe header is used to adjust the index of the sub-image, increasing the processor's computational burden.
Adopt an addressing scheme where each strip is addressed according to an identifier other than the index (such as the sub-image ID), indicating the striped address in the code stream by encoding ID and striped address length, thus eliminating the need for rewriting. The stripe leads, extracts and maps the subcode stream directly to the correct position.
Reduces the use of network, memory and processing resources, avoids the need to rewrite the lead due to each user request, improves coding efficiency and reduces resource consumption.
Smart Images

Figure CN117956190B_ABST
Abstract
Description
[0001] This application is a divisional application. The application number of the original application is 201980087375.7, and the original application date is December 31, 2019. The entire contents of the original application are incorporated into this application by reference.
[0002] Cross-reference to related applications
[0003] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 787,110, filed by FNU Hendry et al. on December 31, 2018, entitled “Explicit Tile Identifier (ID) Signaling,” and claims the benefit of U.S. Provisional Patent Application No. 62 / 883,537, filed by FNU Hendry et al. on August 6, 2019, entitled “Explicit Tile Identifier (ID) Signaling,” the contents of which are incorporated herein by reference. Technical Field
[0004] The present invention relates generally to video coding, and more particularly to address management when extracting sub-pictures from pictures in video coding. Background Art
[0005] Even if a video is relatively short, a large amount of video data is required to describe it, which can cause difficulties when the data is to be streamed or otherwise transmitted across a communication network with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. Since memory resources may be limited, the size of the video may also be an issue when storing the video on a storage device. Video compression devices typically use software and / or hardware on the source side to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received on the destination side by a video decompression device that decodes the video data. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the invention
[0006] In one embodiment, the present invention includes a method implemented in a decoder. The method includes: a receiver of the decoder receives a substream, wherein the substream includes a sub-image in an image, wherein the image is divided into a plurality of slices including a first slice, a parameter set associated with the image and the sub-image, and a slice header associated with the first slice; a processor of the decoder parses the parameter set to obtain an identifier and a slice address length of the first slice; the processor determines a slice address of the first slice from the slice header based on the identifier and the slice address length; the processor decodes the substream to generate a video sequence including the sub-image of the first slice; the processor sends the video sequence including the sub-image for display. In some video decoding systems, slices (also called tile groups) can be addressed according to a set of indices. These indices can start from index 0 at the upper left corner of the image and increase in raster scan order to end with index N at the lower right corner of the image, where N is the number of indices minus 1. This system is suitable for most applications. However, some applications, such as virtual reality (VR), only display sub-images in an image. Some systems can improve coding efficiency by sending only a sub-stream of the codestream to the decoder when streaming VR content, where the sub-stream includes the sub-image to be displayed. In this case, since the upper left corner of the sub-image received by the decoder is usually an index other than 0, the index-based addressing scheme may stop functioning properly. To address these issues, the encoder (or related striper) may need to rewrite each slice header to change the index of the sub-image so that the upper left corner index starts from 0, and the remaining sub-image slices are adjusted accordingly. Dynamically rewriting the slice header (for example, according to each user request) may require the processor to do a lot of calculations. The disclosed system adopts an addressing scheme that can extract the sub-stream including the sub-image without rewriting the slice header. Each slice is addressed according to an identifier (ID) other than the index (for example, a sub-image ID). In this way, no matter which sub-image is received and no matter where the received sub-image is located relative to the upper left corner of the full image, the decoder can consistently determine all relevant addresses. Since the ID is arbitrarily defined (e.g., selected by the encoder), the ID is encoded in a variable length field. Accordingly, the strip address length is also indicated. The ID associated with the sub-image is also indicated. The length is used to resolve the strip address, and the sub-image ID is used to map the strip address from an image-based position to a sub-image-based position. By adopting these mechanisms, encoders, decoders and / or related strips can be improved. For example, a sub-code stream can be extracted and sent instead of extracting and sending the entire code stream, which can reduce the use of network resources, memory resources and / or processing resources.In addition, extracting the sub-code stream in this way does not require rewriting each slice header according to each user request, which further reduces the usage of network resources, memory resources and / or processing resources.
[0007] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the identifier is related to the sub-image.
[0008] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the stripe address length represents the number of bits included in the stripe address.
[0009] Optionally, according to any of the above aspects, in another implementation of the aspect, determining the stripe address of the first stripe includes: the processor uses the length in the parameter set to determine the bit boundary to parse the stripe address from the stripe header; the processor uses the identifier and the stripe address to map the stripe address from an image-based position to a sub-image-based position.
[0010] Optionally, according to any of the above aspects, in another implementation of the aspect, the method further includes: the processor parses the parameter set to obtain an identifier (ID) flag, wherein the ID flag indicates a mapping relationship that can be used to map the strip address from the image-based position to the sub-image-based position.
[0011] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the mapping relationship between the image-based position and the sub-image-based position enables the slice header to be aligned with the sub-image without rewriting the slice header.
[0012] Optionally, in any of the above aspects, in another implementation of the aspect, the stripe address includes a defined value but does not include an index.
[0013] In one embodiment, the present invention includes a method implemented in an encoder. The method includes: a processor of the encoder encodes an image in a bitstream, wherein the image includes a plurality of slices, the plurality of slices including a first slice; the processor encodes a slice header in the bitstream, wherein the slice header includes a slice address of the first slice; the processor encodes a parameter set in the bitstream, wherein the parameter set includes an identifier and a slice address length of the first slice; the processor extracts a sub-codestream from the bitstream by: extracting the first slice according to the slice address of the first slice, the slice address length and the identifier without rewriting the slice header; storing the sub-codestream in a memory of the encoder for transmission to a decoder. In some video decoding systems, slices (also called tile groups) can be addressed according to a set of indices. These indices can start from index 0 at the upper left corner of the image, increase in raster scan order, and end with index N at the lower right corner of the image, where N is the number of indices minus 1. Such a system is suitable for most applications. However, some applications, such as virtual reality (VR), only display sub-images in an image. Some systems can improve coding efficiency by sending only a sub-stream of the codestream to the decoder when streaming VR content, where the sub-stream includes the sub-image to be displayed. In this case, since the upper left corner of the sub-image received by the decoder is usually an index other than 0, the index-based addressing scheme may stop functioning properly. To address these issues, the encoder (or associated striper) may need to rewrite each slice header to change the index of the sub-image so that the upper left corner index starts from 0, and the remaining sub-image slices are adjusted accordingly. Dynamically rewriting the slice header (for example, according to each user request) may require the processor to do a lot of calculations. The disclosed system adopts an addressing scheme that can extract the sub-stream including the sub-image without rewriting the slice header. Each slice is addressed according to an identifier (ID) other than the index (for example, a sub-image ID). In this way, no matter which sub-image is received and no matter where the received sub-image is located relative to the upper left corner of the full image, the decoder can consistently determine all relevant addresses. Since the ID is arbitrarily defined (e.g., selected by the encoder), the ID is encoded in a variable length field. Accordingly, the stripe address length is also indicated. The ID associated with the sub-image is also indicated. The length is used to resolve the stripe address, and the sub-image ID is used to map the stripe address from an image-based location to a sub-image-based location. By adopting these mechanisms, encoders, decoders and / or related slicers can be improved.For example, a substream may be extracted and sent instead of the entire stream, which may reduce the use of network resources, memory resources, and / or processing resources. In addition, extracting the substream may not require rewriting each slice header according to each user request, which further reduces the use of network resources, memory resources, and / or processing resources.
[0014] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the identifier is related to the sub-image.
[0015] Optionally, according to any one of the above aspects, in another implementation manner of the aspect, the stripe address length represents the number of bits included in the stripe address.
[0016] Optionally, according to any of the above aspects, in another implementation of the aspect, the length in the parameter set includes data sufficient to parse the stripe address from the stripe header, and the identifier includes data sufficient to map the stripe address from an image-based position to a sub-image-based position.
[0017] Optionally, according to any of the above aspects, in another implementation of the aspect, the method further includes: the processor encodes an identifier (ID) flag in a parameter set, wherein the ID flag indicates a mapping relationship that can be used to map the strip address from the image-based position to the sub-image-based position.
[0018] Optionally, according to any of the above aspects, in another implementation of the aspect, the stripe address includes a defined value but does not include an index.
[0019] Optionally, according to any one of the above aspects, in another implementation manner of the aspect, extracting the sub-codestream in the codestream includes: extracting a sub-image in the image, wherein the sub-image includes the first stripe, and the sub-codestream includes the sub-image, the stripe header and the parameter set.
[0020] In one embodiment, the present invention includes a video decoding device, which includes: a processor, a memory, a receiver coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the memory, the receiver, and the transmitter are configured to execute the method according to any of the above aspects.
[0021] In one embodiment, the present invention includes a non-transitory computer-readable medium. The non-transitory computer-readable medium includes a computer program product for use by a video decoding device; the computer program product includes computer executable instructions stored in the non-transitory computer-readable medium; when a processor executes the computer executable instructions, the video decoding device performs the method according to any of the above aspects.
[0022] In one embodiment, the present invention includes a decoder. The decoder includes: a receiving module for receiving a sub-code stream, wherein the sub-code stream includes a sub-image in an image, wherein the image is divided into a plurality of stripes including a first stripe, a parameter set related to the image and the sub-image, and a stripe header related to the first stripe; a parsing module for parsing the parameter set to obtain an identifier and a stripe address length of the first stripe; a determining module for determining a stripe address of the first stripe from the stripe header according to the identifier and the stripe address length; a decoding module for decoding the sub-code stream to generate a video sequence including the sub-image of the first stripe; and a sending module for sending the video sequence including the sub-image of the first stripe for display.
[0023] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the decoder is further used to execute the method according to any of the above aspects.
[0024] In one embodiment, the present invention includes an encoder. The encoder includes: an encoding module for: encoding an image in a code stream, wherein the image includes a plurality of slices, and the plurality of slices includes a first slice; encoding a slice header in the code stream, wherein the slice header includes a slice address of the first slice; encoding a parameter set in the code stream, wherein the parameter set includes an identifier and a slice address length of the first slice; an extraction module for extracting a sub-code stream in the code stream by: extracting the first slice according to the slice address of the first slice, the slice address length and the identifier without rewriting the slice header; and a storage module for storing the sub-code stream to be sent to a decoder.
[0025] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the encoder is also used to execute the method according to any of the above aspects.
[0026] For the sake of clarity, any of the above-described embodiments may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the invention.
[0027] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] For a more thorough understanding of the present invention, reference is now made to the following description of the drawings in conjunction with the accompanying drawings and specific embodiments, wherein the same reference numerals represent the same components.
[0029] Figure 1 A flow chart of an exemplary method of encoding a video signal.
[0030] Figure 2 is a schematic diagram of an exemplary encoding and decoding (codec) system for video coding.
[0031] Figure 3 is a schematic diagram of an exemplary video encoder.
[0032] Figure 4 is a schematic diagram of an exemplary video decoder.
[0033] Figure 5 A schematic diagram of an exemplary sub-code stream extracted from a code stream.
[0034] Figure 6 A schematic diagram of segmenting an exemplary image for encoding.
[0035] Figure 7 is a schematic diagram of an exemplary sub-image extracted from an image.
[0036] Figure 8 is a schematic diagram of an exemplary video decoding device.
[0037] Fig. 9 The present invention is a flowchart of an exemplary method for encoding a code stream of an image, wherein the method extracts a sub-code stream including a sub-image without rewriting a slice header through explicit address indication.
[0038] Fig.10 The present invention is a flow chart of an exemplary method for decoding a sub-stream including a sub-image by explicit address indication.
[0039] Fig.11 The present invention is a schematic diagram of an exemplary system for transmitting a sub-stream including a sub-image through explicit address indication. DETAILED DESCRIPTION
[0040] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or existing. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and their full scope of equivalents.
[0041] This document uses various abbreviations such as coding tree block (CTB), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), Joint Video Experts Team (JVET), motion constrained tile set (MCTS), maximum transfer unit (MTU), network abstraction layer (NAL), picture order count (POC), raw byte sequence payload (RBSP), sequence parameter set (SPS), versatile video coding (VVC), and working draft (WD).
[0042] Many video compression techniques can be used to reduce video files while minimizing data loss. For example, video compression techniques can include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in a video sequence. For block-based video decoding, a video strip (e.g., a video image or a portion of a video image) can be divided into video blocks, which can also be referred to as treeblocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in intra-frame coded (I) strips within an image are encoded using spatial predictions for reference samples in adjacent blocks within the same image. Video blocks in inter-frame coded unidirectional prediction (P) or bidirectional prediction (B) strips within an image can be encoded using spatial predictions for reference samples in adjacent blocks within the same image or using temporal predictions for reference samples in other reference images. A picture (picture / image) can be referred to as a frame, and a reference image can be referred to as a reference frame. Spatial prediction or temporal prediction produces a prediction block representing an image block. The residual data represents the pixel difference between the original image block and the predicted block. Accordingly, the inter-coded block is encoded according to the motion vector and the residual data, wherein the motion vector points to the block of reference samples that constitute the predicted block, and the residual data indicates the difference between the coded block and the predicted block. The intra-coded block is encoded according to the intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, thereby generating residual transform coefficients that can then be quantized. The quantized transform coefficients are initially arranged in a two-dimensional array. The quantized transform coefficients can be scanned to generate a one-dimensional vector of transform coefficients. Entropy coding can be applied to achieve further compression. This type of video compression technology is discussed in more detail below.
[0043] To ensure that the encoded video can be correctly decoded, the video is encoded and decoded according to the corresponding video coding standards. Video coding standards include International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC) and other extensions. HEVC includes Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC) and other extensions. The joint video experts team (JVET) of ITU-T and ISO / IEC has begun to develop a video coding standard called Versatile Video Coding (VVC). VVC is included in the Working Draft (WD), which includes JVET-L1001-v7.
[0044] In order to encode a video image, the image is first segmented, and then the resulting segments are encoded in a bitstream. There are various image segmentation schemes. For example, an image can be segmented into regular slices, non-independent slices, tiles, and / or segmented according to Wavefront Parallel Processing (WPP). For simplicity, HEVC restricts the encoder so that only regular slices, non-independent slices, tiles, WPP, and combinations thereof can be used when segmenting slices into CTB groups for video decoding. This segmentation can support maximum transfer unit (MTU) size matching, parallel processing, and reduced end-to-end latency. MTU represents the maximum amount of data that can be sent in a single data packet. If the packet payload exceeds the MTU, the payload is divided into two packets through a process called fragmentation.
[0045] A regular strip, also referred to as a strip, is a portion obtained after the image is divided. It can be reconstructed independently of other regular strips in the same image, but there is still mutual dependence due to the existence of loop filtering operations. Each regular strip is encapsulated in its own network abstraction layer (NAL) unit for transmission. In addition, intra-frame prediction (intra-frame sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across strip boundaries may be disabled to support independent reconstruction. This independent reconstruction supports parallel operations. For example, parallelization based on regular strips reduces inter-processor or inter-core communication. However, since regular strips are independent, each strip is associated with a separate strip header. Since each strip has a strip header bit cost and lacks prediction across strip boundaries, the use of regular strips generates a large amount of coding overhead. In addition, regular strips can be used to support MTU size matching requirements. Specifically, since regular strips are encapsulated in separate NAL units and can be encoded independently, each regular strip needs to be smaller than the MTU in the MTU scheme to avoid the strip being split into multiple data packets. Therefore, the stripe layout within the image will contradict each other in order to achieve parallelization and MTU size matching.
[0046] A non-independent slice is similar to a regular slice, but the slice header is shortened, and the image tree block boundary can be segmented without breaking intra-frame prediction. Accordingly, a non-independent slice can divide a regular slice into multiple NAL units, so that the encoding of the entire regular slice is completed first, and then part of the regular slice is sent out, thereby reducing end-to-end delay.
[0047] A tile is a partition within an image formed by horizontal and vertical boundaries that form tile columns and tile rows. Tiles can be encoded in raster scan order (from right to left, from top to bottom). The scan order of CTB is the order in which the scan is performed within a tile. Accordingly, the CTB within the first tile is first encoded in raster scan order, and then the CTB within the next tile is processed. Similar to conventional slices, tiles break the reliance on intra-frame prediction and entropy decoding. However, tiles may not be included in a single NAL unit, so tiles cannot be used to achieve MTU size matching. Each tile can be processed by one processor / core, and the inter-processor / inter-core communication used for intra-frame prediction between processing units that decode adjacent tiles can be limited to sending shared slice headers (when adjacent tiles are in the same slice) and sharing reconstructed samples and metadata related to loop filtering. When multiple tiles are included in a slice, the entry point byte offset of each tile can be indicated in the slice header in addition to the first entry point offset in the slice. For each slice and partition, at least one of the following conditions must be met: (1) all coding tree blocks in a slice belong to the same partition; (2) all coding tree blocks in a partition belong to the same slice.
[0048] In WPP, the image is partitioned into a single row of CTBs. Entropy decoding and prediction mechanisms can use data from CTBs in other rows. Parallel processing can be achieved by parallel decoding of CTB rows. For example, the current row can be decoded in parallel with the previous row. However, the decoding of the current row is delayed by two CTBs compared to the decoding process of the previous rows. This delay ensures that data related to the CTBs located above and to the right of the current CTB in the current row are available before the current CTB is decoded. When represented graphically, this method looks like a wavefront. This staggered start decoding can be parallelized using as many processors / cores as there are CTB rows included in the image. Since intra-frame prediction is supported between adjacent tree block rows within the image, a large amount of inter-processor / inter-core communication may be required to achieve intra-frame prediction. WPP partitioning does not take into account the NAL unit size. Therefore, WPP does not support MTU size matching. However, conventional stripes can be used in conjunction with WPP, incurring certain coding overhead, so as to achieve MTU size matching as needed.
[0049] The tiles may also include a motion constrained tileset. A motion constrained tileset (MCTS) is a tileset such that the associated motion vectors are restricted to pointing to integer pixel positions inside the MCTS and to fractional pixel positions that only require integer pixel positions inside the MCTS for interpolation. In addition, motion vector candidates derived from blocks outside the MCTS for temporal motion vector prediction are not allowed. In this way, each MCTS can be decoded independently without the need for tiles included in the MCTS. A temporal MCTS supplemental enhancement information (SEI) message can be used to indicate that an MCTS is present in the bitstream and to signal the MCTS. The MCTS SEI message provides supplemental information (detailed as part of the SEI message semantics) that can be used to extract the MCTS sub-codestream to generate a consistent bitstream for the MCTS set. The information includes the number of extraction information sets, each of which defines multiple MCTS sets and includes raw bytes sequence payload (RBSP) bytes of the replacement video parameter set (VPS), sequence parameter set (SPS) and picture parameter set (PPS) to be used in the MCTS substream extraction process. Because one or all of the slice address-related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) can take different values in the extracted substream, the parameter sets (VPS, SPS and PPS) can be rewritten or replaced and the slice header can be updated when extracting the substream according to the MCTS substream extraction process.
[0050] There may be some problems with the above scheme. In some systems, when there are multiple tiles / slices in an image, a syntax element (such as tile_group_address) can be used to indicate the address of the tile group as an index in the tile group header. Tile_group_address represents the tile address of the first tile in the tile group. The length of tile_group_address can be determined as Ceil(Log2(NumTilesInPic)) bits, where NumTilesInPic includes the number of tiles in the image. The value range of tile_group_address can be 0 to NumTilesInPic-1 (including the end value), and the value of tile_group_address may not be equal to the value of tile_group_address of any other coded tile group NAL unit in the same coded image. When tile_group_address does not exist in the bitstream, tile_group_address can be inferred to be 0. The tile address described above includes a tile index. However, using a tile index as the address of each tile group may reduce coding efficiency to some extent.
[0051] For example, some situations may require the modification of the AVC or HEVC slice segment header between encoding and decoding, either directly on the client side or in some network-based media processing entity before sending the sub-stream to the decoder. An example of this situation is tile-based streaming. In tiled streaming, panoramic video is encoded using HEVC tiles, but the decoder only decodes a portion of these tiles. By rewriting the HEVC slice segment header (SSH) and SPS / PPS, the stream can be controlled to change the subset of tiles being decoded and their spatial arrangement in the decoded video frame. One reason for the CPU processing overhead is that the AVC and HEVC slice segment headers use variable length fields and have a byte alignment field at the end. That is, every time a field in the SSH is changed, it affects the byte alignment field at the end of the SSH, which is then rewritten. Moreover, since all fields are variable length encoded, the only way to know the position of the byte alignment field is to parse all the previous fields. This creates a lot of processing overhead, especially when using tiles, where each second of video may include hundreds of NALs. Some systems support explicit indication of a segment identifier (ID). However, some syntax elements may not be optimized and may include unnecessary bits and / or redundant bits when indicated. In addition, some constraints related to explicit segment ID indication are not specified.
[0052] For example, the above mechanism allows the segmentation and compression of an image. For example, an image can be segmented into strips, blocks, and / or groups of blocks. In some examples, groups of blocks can be used interchangeably with strips. Such strips and / or groups of blocks can be addressed according to a set of indices. These indices can start with index 0 at the upper left corner of the image, increase in raster scan order, and end with index N at the lower right corner of the image. In this case, N is the number of indices minus 1. This system is suitable for most applications. However, some applications, such as virtual reality (VR), only display sub-images in an image. Such sub-images may be referred to as regions of interest in some contexts. Some systems only send a sub-stream of the codestream to the decoder when streaming VR content, which can improve coding efficiency, wherein the sub-stream includes the sub-image to be displayed. In this case, since the upper left corner of the sub-image received by the decoder is usually an index other than 0, the index-based addressing scheme may stop functioning properly. To address these issues, the encoder (or associated striper) may need to rewrite each slice header to change the sub-image index so that the top left corner index starts at 0, and adjust the remaining sub-image slices accordingly. Rewriting the slice header dynamically (e.g., upon each user request) may require the processor to do a lot of calculations.
[0053] Various mechanisms are disclosed herein to improve coding efficiency and reduce processing overhead when extracting a substream including a sub-image from an encoded stream including an image. The disclosed system employs an addressing scheme that can extract a substream including a sub-image without rewriting a slice header. Each slice / block group is addressed according to an ID other than an index. For example, a slice can be addressed by a value that can be mapped to an index and stored in a slice header. In this way, the decoder reads the slice address from the slice header and maps the address from a picture-based location to a sub-image-based location. Since the slice address is not a predefined index, the slice address is encoded in a variable length field. Accordingly, the slice address length is also indicated. The ID associated with the sub-image is also indicated. The sub-image ID and length can be indicated in the PPS. A flag can also be indicated in the PPS to indicate that an explicit addressing scheme is adopted. After reading the flag, the decoder can obtain the length and sub-image ID. The length is used to parse the slice address from the slice header. The sub-image ID is used to map the slice address from a picture-based position to a sub-image-based position. In this way, no matter which sub-image is received and no matter where the received sub-image is located relative to the upper left corner of the full image, the decoder can consistently determine all relevant addresses. In addition, this mechanism can make these decisions without rewriting the slice header to change the slice address value and / or without changing the byte alignment field associated with the slice address. By adopting the above mechanism, encoders, decoders and / or related slicers can be improved. For example, sub-code streams can be extracted and sent instead of extracting and sending the entire code stream, which can reduce the use of network resources, memory resources and / or processing resources. In addition, extracting the sub-code stream in this way can avoid rewriting each slice header according to each user request, which further reduces the use of network resources, memory resources and / or processing resources.
[0054] Figure 1 Flowchart of an exemplary method 100 for encoding a video signal. Specifically, an encoder encodes a video signal. During the encoding process, various mechanisms are used to compress the video signal to reduce the size of the video file. The file size is smaller, and the compressed video file can be sent to the user while reducing the associated bandwidth overhead. Then, a decoder decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process is generally the inverse of the encoding process, so that the video signal reconstructed by the decoder is consistent with the video signal on the encoder side.
[0055] In step 101, a video signal is input into an encoder. For example, the video signal may be an uncompressed video file stored in a memory. For another example, a video file may be captured by a video capture device (e.g., a camera) and encoded to support live streaming of the video. The video file may include an audio component and a video component. The video component includes a series of image frames. When these image frames are viewed in sequence, a visual effect of motion is given. These frames include pixels represented by light, referred to herein as brightness components (or brightness samples), and pixels represented by colors, referred to as chrominance components (or color samples). In some examples, these frames may also include depth values to support three-dimensional viewing.
[0056] In step 103, the video is segmented into blocks. Segmentation includes subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), the frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels × 64 pixels). These CTUs include luminance samples and chrominance samples. The CTU can be divided into blocks using a coding tree, and then the blocks are repeatedly subdivided until a configuration that supports further encoding is obtained. For example, the luminance component of the frame can be subdivided until each block includes a relatively uniform luminance value. In addition, the chrominance component of the frame can be subdivided until each block includes a relatively uniform color value. Therefore, the segmentation mechanism varies depending on the content of the video frame.
[0057] In step 105, various compression mechanisms are used to compress the image blocks segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction takes advantage of the fact that objects in common scenes tend to appear in consecutive frames. Therefore, blocks representing objects in a reference frame do not need to be repeatedly represented in adjacent frames. Specifically, an object (such as a table) may remain in a fixed position in multiple frames. Therefore, the table is represented once, and adjacent frames can re-reference the reference frame. A pattern matching mechanism can be used to match objects in multiple frames. In addition, moving objects can be represented in multiple frames due to reasons such as object movement or camera movement. In a particular example, a video can show a car moving across the screen in multiple frames. Motion vectors can be used to represent this movement. A motion vector is a two-dimensional vector that provides the offset between the coordinates of an object in one frame and the coordinates of the object in a reference frame. Therefore, inter-frame prediction can encode image blocks in a current frame as a set of motion vectors that indicate the offset between image blocks in the current frame and corresponding blocks in the reference frame.
[0058] Intra prediction is used to encode blocks in a common frame. Intra prediction takes advantage of the fact that luminance and chrominance components tend to be clustered in a frame. For example, a patch of green in a part of a tree tends to be adjacent to similar patches of green. Intra prediction uses a variety of directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. These directional modes indicate that the samples of the current block are similar / identical to the samples of neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks on a row / column (e.g., a plane) can be interpolated based on neighboring blocks at the edge of the row. Planar mode actually means that light / color smoothly transitions between rows / columns by using a relatively constant slope with a numerical change. The DC mode is used for boundary smoothing, indicating that the block is similar / identical to the average of the samples of all neighboring blocks, which are related to the angular direction of the directional prediction mode. Therefore, the intra prediction block can represent the image block as various relational prediction mode values instead of the actual value. In addition, the inter prediction block can represent the image block as a motion vector value instead of the actual value. In either case, the prediction block may not accurately represent the image block in some cases. All differences are stored in residual blocks. The residual blocks can be transformed to further compress the file.
[0059] In step 107, various filtering techniques can be used. In HEVC, filters are used according to an in-loop filtering scheme. The block-based prediction described above may produce blocky images on the decoder side. In addition, a block-based prediction scheme can encode the block and then reconstruct the encoded block for subsequent use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to the block / frame. These filters reduce block artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference block, making it less likely that the artifact will produce other artifacts in the subsequent block encoded according to the reconstructed reference block.
[0060] Once the video signal is segmented, compressed and filtered, the resulting data is encoded in a bitstream in step 109. The bitstream includes the data described above and any indicative data required to support appropriate reconstruction of the video signal on the decoder side. For example, these data may include segmented portion data, prediction data, residual blocks, and various flags that provide encoding instructions to the decoder. The bitstream can be stored in a memory so that it can be sent to a decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. The generation of the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 can be performed continuously and / or simultaneously in multiple frames and blocks. Figure 1 The order shown is for purposes of clarity and ease of discussion and is not intended to limit the video coding process to a particular order.
[0061] In step 111, the decoder receives the code stream and starts the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the code stream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the code stream to determine the segmented parts of the frame. The segmentation should match the result of the block segmentation in step 103. The entropy encoding / decoding used in step 111 is described below. The encoder makes many choices during the compression process, such as selecting a block segmentation scheme from several possible choices based on the spatial position of the values in one or more input images. A large number of binary symbols (bins) may be used to indicate the exact choice. "Binary symbol" as used herein is a binary value that is a variable (e.g., a bit value that may vary depending on the context). Entropy encoding causes the encoder to discard any options that are obviously not suitable for a particular situation, leaving a set of available options. Then, a codeword is assigned to each available option. The length of the codeword depends on the number of available options (e.g., one binary symbol corresponds to two options, two binary symbols correspond to three to four options, and so on). Then, the encoder encodes the codeword of the selected option. This scheme reduces the codeword because the codeword is as large as expected, uniquely indicating a selection from a small subset of available options, rather than a selection from a potentially large set of all possible options. The decoder then decodes the option by determining the set of available options in a similar manner to the encoder. By determining the set of available options, the decoder can read the codeword and determine the selection made by the encoder.
[0062] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. Then, the decoder uses the residual block and the corresponding prediction block to reconstruct the image block according to the segmentation. The prediction block may include the intra-frame prediction block and the inter-frame prediction block generated by the encoder in step 105. Then, the reconstructed image block is placed in the frame of the reconstructed video signal according to the segmentation data determined in step 111. The syntax for step 113 can also be indicated in the bitstream by entropy coding described above.
[0063] In step 115, filtering is performed on the frame of the reconstructed video signal in a manner similar to step 107 on the encoder side. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter may be applied to the frame to remove blocking artifacts. Once the frame is filtered, in step 117, the video signal may be output to a display for viewing by an end user.
[0064] Figure 21 is a schematic diagram of an exemplary encoding and decoding (codec) system 200 for video decoding. Specifically, the functionality of the codec system 200 can implement the operating method 100. The codec system 200 is generally applicable to describing components used in both encoders and decoders. The codec system 200 receives a video signal and segments the video signal, as described in conjunction with steps 101 and 103 in the operating method 100, to obtain segmented video signals 201. When acting as an encoder, the codec system 200 compresses the segmented video signals 201 into an encoded bitstream, as described in conjunction with steps 105, 107, and 109 in the method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream, as described in conjunction with steps 111, 113, 115, and 117 in the operating method 100. The codec system 200 includes an overall decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header format and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. Figure 2 In FIG. 2 , the black lines represent the movement of the data to be encoded / decoded, and the dotted lines represent the movement of the control data that controls the operation of other components. The components in the codec system 200 may all be present in the encoder. The decoder may include a subset of the components in the codec system 200. For example, the decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.
[0065] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. These blocks can be called nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks may be included in a coding unit (coding unit, CU). For example, a CU may be a sub-part of a CTU, including a luminance block, one or more red difference chrominance (Cr) blocks and one or more blue difference chrominance (Cb) blocks and a syntax instruction corresponding to the CU. The partitioning mode may include a binary tree (BT), a triple tree (TT) and a quad tree (QT) for partitioning a node into two, three or four child nodes of different shapes (depending on the partitioning mode used). The segmented video signal 201 is sent to the overall encoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.
[0066] The overall decoder control component 211 is used to decide whether to encode images in the video sequence in the bitstream according to the application constraints. For example, the overall decoder control component 211 manages the optimization of the bitrate / bitstream size and the reconstruction quality. These decisions can be made based on the storage space / bandwidth availability and the image resolution request. The overall decoder control component 211 also manages the buffer utilization according to the transmission speed to alleviate the buffer underload and overload. To solve these problems, the overall decoder control component 211 manages the segmentation, prediction and filtering performed by other components. For example, the overall decoder control component 211 can dynamically increase the compression complexity to improve the resolution and bandwidth utilization, or reduce the compression complexity to reduce the resolution and bandwidth utilization. Therefore, the overall decoder control component 211 controls other components in the codec system 200 to balance the video signal reconstruction quality and bitrate. The overall decoder control component 211 generates control data, which is used to control the operation of other components. The control data is also sent to the header format and CABAC component 231 to be encoded in the bitstream to indicate the parameters used by the decoder for decoding.
[0067] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices in the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction encoding on the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 can perform multiple decoding passes to select an appropriate encoding mode for each block in the video data, and so on.
[0068] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but for conceptual purposes, they are described separately. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors, which are used to estimate the motion of the video block. For example, a motion vector can represent the displacement of the coded object relative to the prediction block. A prediction block is a block that highly matches the block to be coded in terms of pixel difference. A prediction block can also be referred to as a reference block. This pixel difference can be determined by the sum of absolute difference (SAD), the sum of square difference (SSD), or other difference metrics. HEVC uses several coding objects, including CTU, coding tree block (CTB), and CU. For example, a CTU can be divided into CTBs, and then the CTBs can be divided into CBs for inclusion in a CU. A CU can be encoded as a prediction unit (PU) including prediction data and / or a transform unit (TU) including transform residual data of a CU. The motion estimation component 221 performs a rate-distortion analysis as part of a rate-distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and can select the reference block, motion vector, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics can maintain a balance between the quality of video reconstruction (e.g., the amount of data loss caused by compression) and the coding efficiency (e.g., the size of the final code).
[0069] In some examples, the codec system 200 can calculate the value of the sub-integer pixel position of the reference image stored in the decoded image buffer component 223. For example, the video codec system 200 can interpolate the quarter pixel position, the eighth pixel position, or other fractional pixel position of the reference image. Therefore, the motion estimation component 221 can perform a motion search relative to the integer pixel position and the fractional pixel position, and output a motion vector with fractional pixel accuracy. The motion estimation component 221 compares the position of the PU with the position of the prediction block in the reference image and calculates the motion vector for the PU of the video block in the inter-frame coded slice. The motion estimation component 221 outputs the calculated motion vector as motion data to the header format and CABAC component 231 for encoding, and outputs it as motion data to the motion compensation component 219.
[0070] The motion compensation performed by the motion compensation component 219 may include obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. Likewise, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. After receiving the motion vector of the PU of the current video block, the motion compensation component 219 may locate the prediction block pointed to by the motion vector. The pixel values of the prediction block are then subtracted from the pixel values of the current video block being encoded to obtain a pixel difference, thereby forming a residual video block. In general, the motion estimation component 221 performs motion estimation relative to the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for the chroma component and the luma component. The prediction block and the residual block are sent to the transform scaling and quantization component 213.
[0071] The segmented video signal 201 is also sent to the intra-frame estimation component 215 and the intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but for conceptual purposes, they are described separately. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction on the current block relative to the blocks in the current frame, replacing the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines an intra-frame prediction mode to encode the current block. In some examples, the intra-frame estimation component 215 selects a suitable intra-frame prediction mode from a plurality of tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then sent to the header format and CABAC component 231 for encoding.
[0072] For example, the intra-frame estimation component 215 performs rate-distortion analysis on various tested intra-frame prediction modes, calculates rate-distortion values, and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis is generally used to determine the amount of distortion (or error) between the encoded block and the original uncoded block encoded to generate the encoded block, and to determine the bit rate (e.g., number of bits) used to generate the encoded block. The intra-frame estimation component 215 calculates ratios based on the distortion and rate of the various encoded blocks to determine the intra-frame prediction mode that obtains the best rate-distortion value for the block. In addition, the intra-frame estimation component 215 can be used to encode the depth block in the depth image using the depth modeling mode (DMM) according to rate-distortion optimization (RDO).
[0073] When implemented on an encoder, the intra prediction component 217 can generate a residual block based on the prediction block according to the intra prediction mode determined by the intra estimation component 215, or read the residual block from the bitstream when implemented on a decoder. The residual block includes the difference between the predicted block and the original block, represented as a matrix. The residual block is then sent to the transform scaling and quantization component 213. The intra estimation component 215 and the intra prediction component 217 can operate on the luma component and the chroma component.
[0074] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 performs a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform on the residual block to generate a video block including residual transform coefficient values. Wavelet transform, integer transform, subband transform or other types of transforms can also be performed. The transform can convert the residual information from the pixel value domain to the transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also used to scale the transform residual information according to frequency, etc. This scaling involves applying a scaling factor to the residual information so that different frequency information is quantized at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be changed by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 can then perform a scan on the matrix including the quantized transform coefficients. The quantized transform coefficients are sent to the header format and CABAC component 231 to be encoded in the codestream.
[0075] The scaling and inverse transform component 229 performs operations opposite to the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 performs inverse scaling, inverse transform and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, for subsequent use as a reference block. The reference block may become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 may add the residual block back to the corresponding prediction block to calculate the reference block for motion estimation of subsequent blocks / frames. A filter is applied to the reconstructed reference block to reduce artifacts generated during scaling, quantization and transformation. These artifacts may make the prediction inaccurate (and generate additional artifacts) when predicting subsequent blocks.
[0076] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block from the scaling and inverse transform component 229 can be combined with the corresponding prediction block from the intra-frame prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. Then, the filter can be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block. Figure 2 Like other components in the, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and can be implemented together, but for conceptual purposes, they are described separately. The filters applied to the reconstructed reference block are applied to specific spatial regions, and these filters include multiple parameters to adjust the way these filters are used. The filter control analysis component 227 analyzes the reconstructed reference block to determine the locations where these filters need to be used and sets the corresponding parameters. This data is sent to the header format and CABAC component 231 and encoded as filter control data. The in-loop filter component 225 uses these filters according to the filter control data. These filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. These filters can be applied in the spatial domain / pixel domain (e.g., reconstructed pixel blocks) or the frequency domain according to the examples.
[0077] When operating as an encoder, the filtered reconstructed image blocks, residual blocks and / or prediction blocks are stored in the decoded image buffer component 223 for subsequent use in motion estimation as described above. When operating as a decoder, the decoded image buffer component 223 stores the filtered reconstructed blocks and sends them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks and / or reconstructed image blocks.
[0078] The header format and CABAC component 231 receives data from various components in the codec system 200 and encodes the data in an encoded bitstream to be sent to the decoder. Specifically, the header format and CABAC component 231 generates various headers to encode control data (such as overall control data and filter control data). In addition, prediction data (including intra-frame prediction and motion data) and residual data in the form of quantized transform coefficient data are encoded in the bitstream. The final bitstream includes all the information required by the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword mapping table), definitions of the coding contexts of various blocks, indications of the most likely intra-frame prediction mode, indications of segmentation information, etc. This data can be encoded using entropy coding. For example, the above information may be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding or other entropy coding techniques. After entropy coding, the encoded bitstream may be sent to another device (e.g., a video decoder) or archived for subsequent transmission or retrieval.
[0079] Figure 3 is a block diagram of an exemplary video encoder 300. The video encoder 300 may be used to implement the encoding function of the codec system 200 and / or perform steps 101, 103, 105, 107, and / or 109 in the operation method 100. The encoder 300 segments the input video signal to obtain segmented video signals 301 that are substantially similar to the segmented video signals 201. Then, the components in the encoder 300 compress and encode the segmented video signals 301 in a bitstream.
[0080] Specifically, the segmented video signal 301 is sent to the intra prediction component 317 for intra prediction. The intra prediction component 317 can be substantially similar to the intra estimation component 215 and the intra prediction component 217. The segmented video signal 301 is also sent to the motion compensation component 321 for inter prediction based on the reference block in the decoded image buffer component 323. The motion compensation component 321 can be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction block and the residual block from the intra prediction component 317 and the motion compensation component 321 are sent to the transform and quantization component 313 to transform and quantize the residual block. The transform and quantization component 313 can be substantially similar to the transform scaling and quantization component 213. The transform quantized residual block and the corresponding prediction block (together with the associated control data) are sent to the entropy coding component 331 for encoding in the code stream. The entropy coding component 331 can be substantially similar to the header format and CABAC component 231.
[0081] The transformed quantized residual block and / or the corresponding prediction block are also sent from the transform and quantization component 313 to the inverse transform and quantization component 329 to be reconstructed as a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to an example, the in-loop filter in the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 can include a plurality of filters, as described in conjunction with the in-loop filter component 225. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block for the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.
[0082] Figure 4 4 is a block diagram of an exemplary video decoder 400. The video decoder 400 may be used to implement the decoding function of the codec system 200 and / or perform step 111, step 113, step 115 and / or step 117 in the operation method 100. The decoder 400 receives a bitstream from the encoder 300, etc., and generates a reconstructed output video signal according to the bitstream for display to an end user.
[0083] The code stream is received by an entropy decoding component 433. The entropy decoding component 433 is used to perform an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE encoding or other entropy encoding techniques. For example, the entropy decoding component 433 can use header information to provide context to parse other data encoded in the code stream as code words. The above-mentioned decoding information includes any information required to decode the video signal, such as overall control data, filter control data, segmentation information, motion data, prediction data and quantized transform coefficients in the residual block. The quantized transform coefficients are sent to the inverse transform and quantization component 429 to be reconstructed as a residual block. The inverse transform and quantization component 429 can be similar to the inverse transform and quantization component 329.
[0084] The reconstructed residual block and / or prediction block is sent to the intra prediction component 417 to be reconstructed into an image block according to the intra prediction operation. The intra prediction component 417 can be similar to the intra estimation component 215 and the intra prediction component 217. Specifically, the intra prediction component 417 uses the prediction mode to locate the reference block in the frame, and applies the residual block to the above result to reconstruct the intra prediction image block. The above-mentioned reconstructed intra prediction image block and / or residual block and the corresponding inter prediction data are sent to the decoded image buffer component 423 through the in-loop filter component 425. The decoded image buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the above-mentioned reconstructed image block, residual block and / or prediction block. This information is stored in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is sent to the motion compensation component 421 for inter prediction. The motion compensation component 421 can be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses the motion vector of the reference block to generate a prediction block, and applies the residual block to the above result to reconstruct the image block. The resulting reconstructed block can also be sent to the decoded image buffer component 423 through the in-loop filter component 425. The decoded image buffer component 423 continues to store other reconstructed image blocks. These reconstructed image blocks can be reconstructed into frames through segmentation information. These frames can also be placed in a sequence. The sequence is output to the display as a reconstructed output video signal.
[0085] Figure 5 is a schematic diagram of an exemplary bitstream 500 including an encoded video sequence. For example, the bitstream 500 may be generated by the codec system 200 and / or the encoder 300, and decoded by the codec system 200 and / or the decoder 400. For another example, the bitstream 500 may be generated by the encoder in step 109 of the method 100, and used by the decoder in step 111.
[0086] The code stream 500 includes a sequence parameter set (SPS) 510, a plurality of picture parameter sets (PPS) 512, a plurality of slice headers 514, and image data 520. The SPS 510 includes sequence data common to all images in the video sequence included in the code stream 500. These data may include image size, bit depth, coding tool parameters, bit rate constraints, etc. The PPS 512 includes one or more parameters specific to the corresponding image. Therefore, each image in the video sequence can refer to a PPS 512. The PPS 512 can indicate the coding tools, quantization parameters, offsets, image-specific coding tool parameters (such as filter control parameters) that can be used for the partitions in the corresponding image. The slice header 514 includes one or more parameters specific to the corresponding slices in the image. Therefore, each slice in the video sequence can refer to the slice header 514. The slice header 514 may include slice type information, picture order count (POC), reference picture list, prediction weight, partition entry point, deblocking filter parameters, etc. In some examples, a slice may be referred to as a block group. In this case, the slice header 514 may be referred to as a block group header.
[0087] The image data 520 includes video data encoded according to inter-frame prediction and / or intra-frame prediction and corresponding transformed and quantized residual data. These image data 520 are sorted according to the partitioning mode of the image before encoding. For example, the video sequence is divided into images 521, and the image 521 is divided into strips 523. The strips 523 can be further divided into blocks and / or CTUs, and the CTUs are further divided into coding blocks according to the coding tree. Then, the coding blocks can be encoded / decoded according to the prediction mechanism. For example, the image 521 may include one or more strips 523. The image 521 refers to the PPS 512, and the strips 523 refer to the strip header 514. Each strip 523 may include one or more blocks. Each strip 523 and / or the image 521 may include multiple CTUs.
[0088] Each image 521 may include the entire visual data set associated with the video sequence at the corresponding moment. The VR system can display the area selected by the user in the image 521. In this way, a feeling of being in the scene depicted by the image 521 is created. When the code stream 500 is encoded, the area that the user may want to view is unknown. Therefore, the image 521 may include every possible area that the user is likely to view. However, in a VR scenario, the corresponding codec can be designed based on the assumption that the user only views the area selected in the image 521 and discards the rest of the image 521.
[0089] Each strip 523 may be a rectangle defined by an upper left CTU and a lower right CTU. In some examples, strip 523 includes a series of blocks and / or CTUs that are raster scanned in a left-to-right, top-to-bottom order. In other examples, strip 523 is a rectangular strip. A rectangular strip may not traverse the entire width of the image in a raster scan order. Instead, a rectangular strip may include a rectangular and / or square area defined in an image 521 according to CTU and / or block rows and CTU and / or block columns. Strip 523 is the smallest unit that a decoder can display separately. Therefore, strips 523 in image 521 can be assigned to different sub-images 522 to respectively depict desired areas of image 521. For example, in a VR scene, image 521 may include the entire visible data range, but a user can only view sub-images 522 including one or more strips 523 on a head-mounted display.
[0090] As described above, the video codec may assume that non-selected areas of the image 521 are to be discarded at the decoder side. Therefore, a sub-stream 501 may be extracted from the codestream 500. The extracted sub-stream 501 may include a selected sub-image 522 and associated syntax. Non-selected areas of the image 521 may be sent at a lower resolution or ignored to improve coding efficiency. The sub-image 522 is a selected area of the image 521 and may include one or more associated slices 524. The slice 524 is a subset of the slice 523 and depicts a selected area of the image 521 associated with the sub-image 522. The sub-stream 501 also includes an SPS 510, a PPS 512, a slice header 514, and / or sub-portions thereof associated with the sub-image 522 and the slice 524.
[0091] Substreams 501 may be extracted from codestream 500. For example, a user using a decoder may watch a video. The user may select a corresponding region of an image 521. The decoder may request a subsequent sub-image 522 associated with the region currently being viewed by the user. The encoder may then send the sub-image 522 associated with the selected region at a higher resolution and the remaining region of the image 521 at a lower resolution. To achieve this functionality, the decoder may extract 529 one or more substreams 501 from codestream 500. Extraction 529 includes storing the sub-image 522 (including the slice 524 in the sub-image 522) in the substream 501. Extraction 529 also includes storing the associated SPS 510, PPS 512, and slice header 514 in the substream as needed to support decoding of the sub-image 522 and slice 524.
[0092] One problem with extracting 529 the substream 501 is that the addressing relative to the image 521 may be different from the addressing relative to the subimage 522. The addressing problem is described in detail below. In some systems, the slice header 514 may be rewritten to adjust for these addressing differences. However, the substream 501 may include many slice headers 514 (for example, one or two orders of magnitude for each image 521), and these slice headers 514 are dynamically rewritten for each user. Therefore, rewriting the slice header 514 in this way may require the processor to do a lot of calculations. The present invention includes a mechanism that can extract 529 the slice header 514 into the substream 501 without rewriting the slice header 514.
[0093] In a system that rewrites the strip header 514, strips 523 and strips 524 are addressed according to index values such as strip index, block index, CTU index, etc. The values of these indexes increase in raster scan order. In order to solve the addressing mismatch problem, the disclosed embodiment uses ID values defined for each strip, block and / or CTU. These defined IDs can be default values and / or can be selected by the encoder. The defined IDs can be uniformly increased in raster scan order, but these defined IDs may not be monotonically increasing. Therefore, there may be value gaps in the defined IDs to achieve address management. For example, the index can be monotonically increasing (e.g., 0, 1, 2, 3, etc.), and the defined ID can be increased by a certain defined multiple (e.g., 0, 10, 20, 30, etc.). The encoder can carry the mapping relationship 535 in the code stream 500 and the sub-code stream 501, so that the decoder can map the defined ID to an index that the decoder can parse.
[0094] The parameter set such as SPS 510 and / or PPS 512 may include an ID flag 531. The ID flag 531 may be set to indicate that a mapping relationship 535 may be used to map a slice address from a position based on a picture 521 to a position based on a sub-picture 522. Accordingly, the ID flag 531 may be set to indicate to the decoder the public mechanism used in the codestream 500 and the sub-codestream 501. For example, the ID flag 531 may be encoded as an explicit block ID flag, sps_subpic_id_present_flag, or other syntax elements. The ID flag 531 may be encoded in the codestream 500 and extracted 529 into the sub-codestream 501.
[0095] The parameter set such as SPS 510 and / or PPS 512 may also include a syntax element ID 532. The ID 532 may indicate a sub-image 522 in the image 521. For example, some IDs 532 may be included in the PPS 512 of the codestream 500. When the sub-codestream 501 is extracted 529, one or more IDs 532 associated with one or more sub-images 522 to be sent to the decoder may be included in the PPS 512 of the sub-codestream 501. In other examples, a marker (point) pointing to the relevant ID 532 may be inserted into the PPS 512 of the sub-codestream 501 so that the decoder can determine the correct ID 532. For example, the ID 532 may be encoded as SubPicIdx, Tile_id_val[i], or other syntax elements indicating the sub-image 522 boundary.
[0096] The parameter set such as SPS 510 and / or PPS 512 may also include a syntax element slice address length 533. In addition, the slice header 514 may include a slice address 534 of the slice 523. The slice address 534 is included as a defined ID value. The slice address 534 may be directly extracted 529 into the slice header 514 of the substream 501 without modification to avoid rewriting the slice header 514. For example, the slice address 534 may be encoded as a slice_address, a tile_group_address, or other syntax elements indicating the slice 523 boundary and the slice 524 boundary. The slice address length 533 may then be used to parse the slice address 534. For example, the slice address 534 includes a value defined by the encoder and is therefore encoded as a variable length value before the byte alignment field. The slice address length 533 may indicate the number of bits included in the corresponding slice address 534, and thus the boundaries of the slice address 534 may be indicated to the decoder. Therefore, the decoder can use the slice address length 533 (e.g., from the PPS 512) to parse the slice address 534. Therefore, the slice header 514 does not need to be rewritten to adjust the byte alignment field after the slice address 534. For example, the slice address length 533 can be encoded as subpic_id_len_minus1, tile_id_len_minus1, or other syntax elements indicating the slice address length 533. The slice address length 533 can be included in the PPS 512 of the codestream 500 and then extracted 529 into the PPS 512 of the sub-codestream 501.
[0097] The mapping relationship 535 may also be sent via a parameter set such as the SPS 510, the PPS 512, and / or the slice header 514. The mapping relationship 535 indicates a mechanism for mapping a slice address from a position based on the picture 521 to a position based on the sub-picture 522. The mapping relationship 535 may be encoded in the codestream 500 and extracted 529 into a corresponding parameter set of the sub-codestream 501. For example, the mapping relationship 535 may be encoded as a syntax element SliceSubpicToPicIdx[SubPicIdx][slice_address], a syntax element tileIdToIdx[Tile_group_address], or other syntax elements indicating a mechanism for mapping a slice address from a position based on the picture 521 to a position based on the sub-picture 522.
[0098] Accordingly, the decoder can read the sub-code stream 501 and obtain the ID tag 531 to determine that the slice 524 is addressed by a defined address rather than an index. The decoder can obtain the ID 532 to determine the sub-image 522 included in the sub-code stream 501. The decoder can also obtain one or more slice addresses 534 and a slice address length 533 to parse the one or more slice addresses 534. Then, the decoder can obtain the mapping relationship 53 to map the one or more slice addresses 534 to a format that the decoder can parse. Finally, the decoder can use the one or more slice addresses 534 when decoding and displaying the sub-image 522 and the corresponding slice 524.
[0099] Figure 6 Schematic diagram of an exemplary image 600 segmented for encoding. For example, the image 600 may be encoded in a code stream 500 and decoded from the code stream 500 by the encoding and decoding system 200, the encoder 300 and / or the decoder 400. In addition, the image 600 may be segmented into and / or included in sub-images of a sub-code stream 501 to implement encoding and decoding in the method 100.
[0100] Image 600 may be partitioned into slices 623, which may be substantially similar to slice 523. Slice 623 may be further partitioned into blocks 625 and CTUs 627. Figure 6 623 are represented by bold lines, with each stripe 623 being graphically distinguished by alternating white backgrounds and hashing. Blocks 625 are represented by dashed lines. Block 625 boundaries that are located on a stripe 623 boundary are shown as bold dashed lines, while block 625 boundaries that are not located on a stripe 623 boundary are shown as thin dashed lines. CTU 627 boundaries are shown as thin solid lines, except for CTU 627 boundaries that are covered by a block 625 boundary or a stripe 623 boundary. In this example, the image 600 includes 9 stripes 623, 24 blocks 625, and 216 CTUs 627.
[0101] As shown, a strip 623 is a rectangle with boundaries that may be defined by a contained block 625 and / or CTU 627. A strip 623 may not span the entire width of the image 600. Blocks 625 may be generated in strips 623 based on rows and columns. CTUs 627 may be segmented from blocks 625 and / or strips 623 to produce segmented portions of the image 600 that may be subdivided into coding blocks for encoding based on inter-frame prediction and / or intra-frame prediction. The image 600 may be encoded in a code stream such as code stream 500. Regions of the image 600 may be included in sub-images and extracted into sub-code streams such as sub-images 522 and sub-code streams 501.
[0102] Figure 7 is a schematic diagram of an exemplary sub-image 722 extracted from the image 700. For example, the image 700 may be substantially similar to the image 600. In addition, the image 700 may be encoded in the code stream 500 by the codec system 200 and / or the encoder 300, etc. The sub-image 722 may be extracted into the sub-code stream 501 and decoded from the sub-code stream 501 by the codec system 200, the encoder 300, and / or the decoder 400, etc. In addition, the image 700 may be used to implement the encoding and decoding in the method 100.
[0103] As shown, the image 700 includes an upper left corner 702 and a lower right corner 704. The sub-image 722 includes one or more strips 723 in the image 700. When using an index, the upper left corner 702 and the lower right corner 704 are respectively related to the first index and the last index. However, the decoder can only display the sub-image 722, but not the entire image 700. In addition, the strip address 734 of the first strip 723a may not be aligned with the upper left corner 702, and the strip address 734 of the third strip 723c may not be aligned with the lower right corner 704. Therefore, the strip address 734 relative to the sub-image 722 is not aligned with the strip address 734 relative to the image 700. The present invention uses an ID defined for the strip address 734 instead of an index. The decoder can use a mapping relationship to map the strip address 734 from a position based on the image 700 to a position based on the sub-image 722. The decoder may then use the mapped stripe addresses 734 to place the first stripe 723a at the upper left corner 702 of the decoder display, the third stripe 723c at the lower left corner 704 of the decoder display, and the second stripe 723b between the first stripe 723a and the third stripe 723c.
[0104] As described herein, the present invention describes improvements to explicit tile ID indication in video decoding, where tiles are used for image segmentation. The above technical description is based on VVC developed by JVET of ITU-T and ISO / IEC. However, these techniques are also applicable to other video codec specifications. The following is an exemplary embodiment described herein.
[0105] The concepts of block index and block ID can be distinguished. The block ID of a block may be equal to or not equal to the block index of the block. When the block ID is different from the block index, the mapping relationship between the block ID and the block index can be indicated in the PPS. The block ID can be used to indicate the block group address in the block group header instead of using the block index. In this way, the value of the block ID can remain the same when the block group is extracted from the original code stream. This can be achieved by updating the mapping relationship between the block ID and the block index in the PPS referenced by the block group. This method solves the problem that the value of the block index may change depending on the sub-image to be extracted. It should be noted that when performing MCTS-based sub-code stream extraction, other parameter sets (such as parameter sets other than the slice header) may still need to be rewritten.
[0106] The above can be implemented by using a flag in a parameter set indicating tile information. For example, PPS can be used as a parameter set. For example, explicit_tile_id_flag can be used for this purpose. No matter how many tiles there are in the image, explicit_tile_id_flag can be indicated, and explicit_tile_id_flag can indicate that an explicit tile indication is used. Syntax elements can also be used to indicate the number of bits used for a tile ID value indication (e.g., a mapping relationship between a tile index and a tile ID). Such syntax elements can also be used to indicate a tile ID / address in a tile group header. For example, the syntax element tile_id_len_minus1 can be used for this purpose. When explicit_tile_id_flag is equal to 0 (e.g., when the tile ID is set to a tile index), tile_id_len_minus1 may not exist. When tile_id_len_minus1 does not exist, it can be inferred that the value of tile_id_len_minus1 is equal to the value of Ceil(Log2(NumTilesInPic)). Another constraint would require that the bitstream resulting from MCTS substream extraction may include explicit_tile_id_flag set to 1 for the active PPS unless the substream includes the top left tile in the original bitstream.
[0107] In an exemplary embodiment, the video coding syntax may be modified as described below to implement the functionality described herein. An exemplary CTB raster and block scanning process may be described as follows. The list TileId[ctbAddrTs] (the value range of ctbAddrTs is 0 to PicSizeInCtbsY–1, including the end value) represents the conversion of the CTB address to the block ID under block scanning, and the list NumCtusInTile[tileIdx] (the value range of tileIdx is 0 to PicSizeInCtbsY–1, including the end value) represents the conversion of the block index to the number of CTUs in the block, and the two are derived as follows:
[0108]
[0109] The list NumCtusInTile[tileIdx] (tileIdx ranges from 0 to PicSizeInCtbsY–1, including the end value) represents the conversion from tile index to the number of CTUs in the tile, which can be derived as follows:
[0110]
[0111] The set TileIdToIdx[tileId] for a set of NumTilesInPic tileId values represents the conversion from tile ID to tile index, which can be derived as follows:
[0112]
[0113]
[0114] An exemplary picture parameter set RBSP syntax may be described as follows.
[0115]
[0116]
[0117] An exemplary chunk group header syntax may be described as follows.
[0118]
[0119] An exemplary chunk group data syntax may be described as follows.
[0120]
[0121] The semantics of an exemplary picture parameter set RBSP can be described as follows. The explicit_tile_id_flag is set to 1, indicating that the tile ID of each tile is explicitly indicated. The explicit_tile_id_flag is set to 0, indicating that the tile ID is not explicitly indicated. For a bitstream that is a result of MCTS substream extraction, the value of explicit_tile_id_flag can be set to 1 for an activated PPS unless the resulting bitstream includes the top left tile in the original bitstream. Tile_id_len_minus1+1 represents the number of bits of the syntax element tile_id_val[i] and the syntax element tile_group_address in the tile group header used to represent the reference PPS. The value range of tile_id_len_minus1 can be Ceil(Log2(NumTilesInPic)~15 (including the end value). When tile_id_len_minus1 does not exist, it can be inferred that the value of tile_id_len_minus1 is equal to Ceil(Log2(NumTilesInPic). It should be noted that in some cases, the value of tile_id_len_minus1 can be greater than Ceil(Log2(NumTilesInPic). This is because the current code stream may be the result of MCTS sub-code stream extraction. At this time, the tile ID can be the tile index in the original code stream, and Ceil(Log2(OrgNumTilesInPic) can be used. Pic) bits, where OrgNumTilesInPic is the NumTilesInPic of the original bitstream, which is greater than the NumTilesInPic of the current bitstream. tile_id_val[i] represents the tile ID of the i-th tile in the image of the reference PPS. The length of tile_id_val[i] is (tile_id_len_minus1+1) bits. For any integers m and n in the range 0 to NumTilesInPic–1 (including the end values), when m is not equal to n, tile_id_val[m] may not be equal to tile_id_val[n], and when m is less than n, tile_id_val[m] may be less than tile_id_val[n].
[0122] The following variables can be derived by calling CTB raster and block scan conversion: the list ColWidth[i] (i ranges from 0 to num_tile_columns_minus1, inclusive) represents the width of the ith block column, in CTBs; the list RowHeight[j] (j ranges from 0 to num_tile_rows_minus1, inclusive) represents the height of the jth block row, in CTBs; the list ColBd[i] (i ranges from 0 to num_tile_columns_minus1+1, inclusive) represents the position of the ith block column boundary. The list RowBd[j] (the value range of j is 0 to num_tile_rows_minus1+1, including the end value) represents the position of the j-th tile row boundary, in CTB units; the list CtbAddrRsToTs[ctbAddrRs] (the value range of ctbAddrRs is 0 to PicSizeInCtbsY–1, including the end value) represents the conversion of the CTB address under the CTB raster scan of the image to the CTB address under the block scan; the list CtbAddrTsToRs[ctbAddrTs] (the value range of ctbAddrTs is 0 to PicSize InCtbsY–1, including the end value) represents the conversion of the CTB address under block scanning to the CTB address under the CTB raster scanning of the image; the list TileId[ctbAddrTs] (the value range of ctbAddrTs is 0 to PicSizeInCtbsY–1, including the end value) represents the conversion of the CTB address under block scanning to the block ID; the list NumCtusInTile[tileIdx] (the value range of tileIdx is 0 to PicSizeInCtbsY–1, including the end value) represents the conversion of the block index to the number of CTUs in the block; the list FirstCtbAddrTs[ti leIdx] (the value range of tileIdx is 0 to NumTilesInPic–1, including the end value) represents the conversion of tile ID to the CTB address of the first CTB in the tile under tile scanning; the set TileIdToIdx[tileId] for a set of NumTilesInPic tileId values represents the conversion of tile ID to tile index, and the list FirstCtbAddrTs[tileIdx] (the value range of tileIdx is 0 to NumTilesInPic–1, including the end value) represents the conversion of tile ID to the CTB address of the first CTB in the tile under tile scanning;The list ColumnWidthInLumaSamples[i] (i ranges from 0 to num_tile_columns_minus1, including the end value) represents the width of the i-th tile column, in units of luma samples; the list RowHeightInLumaSamples[j] (j ranges from 0 to num_tile_rows_minus1, including the end value) represents the height of the j-th tile row, in units of luma samples. ;
[0123] tile_group_address indicates the tile ID of the first tile in the tile group. The length of tile_group_address is (tile_id_len_minus1+1) bits. The value of tile_group_address can range from 0 to 2tile_id_len_minus1+1–1 (including the end value), and the value of tile_group_address may not be equal to the value of tile_group_address of any other coded tile group NAL unit in the same coded picture.
[0124] Figure 8 8 is a schematic diagram of an exemplary video decoding device 800. The video decoding device 800 is suitable for implementing the disclosed examples / embodiments described herein. The video decoding device 800 includes a downstream port 820, an upstream port 850 and / or a transceiver unit (Tx / Rx) 810. The transceiver unit 810 includes a transmitter and / or a receiver for communicating data in the upstream and / or downstream through a network. The video decoding device 800 also includes a processor 830 and a memory 832. The processor 830 includes a logic unit and / or a central processing unit (CPU) to process data. The memory 832 is used to store the data. The video decoding device 800 may also include an electrical component, an optical-to-electrical (OE) component, an electrical-to-optical (EO) component, and / or a wireless communication component coupled to the upstream port 850 and / or the downstream port 820 for communicating data through an electrical communication network, an optical communication network, or a wireless communication network. The video decoding device 800 may also include an input and / or output (I / O) device 860 for communicating data with a user. The I / O device 860 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 860 may also include an input device such as a keyboard, a mouse, a trackball, etc. and / or a corresponding interface for interacting with the above-mentioned output devices.
[0125] The processor 830 is implemented by hardware and software. The processor 830 can be implemented as one or more CPU chips, one or more cores (for example, implemented as a multi-core processor), one or more field-programmable gate arrays (FPGA), one or more application specific integrated circuits (ASIC), and one or more digital signal processors (DSP). The processor 830 communicates with the downstream port 820, the transceiver unit 810, the upstream port 850, and the memory 832. The processor 830 includes a decoding module 814. The decoding module 814 implements the disclosed embodiments described herein, such as method 100, method 900, and / or method 1000, which can use code stream 500, image 600, and / or image 700. The decoding module 814 can also implement any other method / mechanism described herein. In addition, the decoding module 814 can implement the codec system 200, the encoder 300, and / or the decoder 400. For example, when acting as an encoder, the decoding module 814 can indicate a flag, a sub-image ID, and a length in the PPS. The decoding module 814 can also encode the slice address in the slice header. Then, the decoding module 814 can extract the sub-stream including the sub-image from the code stream of the image without rewriting the slice header. When acting as a decoder, the decoding module 814 can read a flag to determine whether to use an explicit slice address instead of an index. The decoding module 814 can also read the length and sub-image ID from the PPS and the slice address from the slice header. Then, the decoding module 814 can use the length to parse the slice address and use the sub-image ID to map the slice address from an image-based address to a sub-image-based address. Therefore, the decoding module 814 can determine the expected position of the slice without considering the selected sub-image, and there is no need to rewrite the slice header to accommodate the sub-image-based address change. Therefore, when segmenting and encoding video data, the decoding module 814 enables the video decoding device 800 to provide other functions, avoid certain processing to reduce processing overhead and / or improve coding efficiency. Accordingly, the decoding module 814 improves the functionality of the video decoding device 800 while solving problems specific to the field of video encoding. In addition, the decoding module 814 affects the transition of the video decoding device 800 to different states. Alternatively, the encoding module 814 can be implemented as instructions stored in the memory 832 and executed by the processor 830 (e.g., implemented as a computer program product stored in a non-transitory medium).
[0126] The memory 832 includes one or more memory types, such as a disk, a tape drive, a solid-state hard disk, a read only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 832 can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the program.
[0127] Fig. 9 A flowchart of an exemplary method 900 for encoding a code stream (e.g., code stream 500) of an image (e.g., image 600) to support extraction of a sub-code stream (e.g., sub-code stream 501) including a sub-image (e.g., sub-image 522) without rewriting a slice header through explicit address indication. The method 900 may be performed by an encoder such as the codec system 200, the encoder 300, and / or the video decoding device 800 when performing the method 100.
[0128] Method 900 may start with: an encoder receiving a video sequence including a plurality of images, and determining to encode the video sequence in a bitstream according to user input, etc. The video sequence is segmented into pictures / images / frames for further segmentation before encoding. In step 901, an image in the video sequence is encoded in a bitstream. The image may include a plurality of strips, and the plurality of strips include a first strip. The first strip may be any strip in the image, but for clarity of discussion, it is described as the first strip. For example, the upper left corner of the first strip may not be aligned with the upper left corner of the image.
[0129] In step 903, a slice header associated with the slice is encoded in the bitstream. The slice header includes a slice address of the first slice. The slice address may include a defined value, such as a value selected by the encoder. Such a value may be arbitrary, but may increase in raster scan order (e.g., from left to right and from top to bottom) to support consistent encoding functionality. The slice address may not include an index. In some examples, the slice address may be a syntax element slice_address.
[0130] In step 905, a PPS is encoded in the codestream. An identifier and a slice address length of the first slice may be encoded in the PPS of the codestream. The identifier may be a sub-image identifier. The slice address length may indicate the number of bits included in the slice address. For example, the slice address length in the PPS may include data sufficient to parse the slice address from the slice header (encoded in step 903). In some examples, the length may be a syntax element subpic_id_len_minus1. In addition, the identifier may include data sufficient to map the slice address from a picture-based position to a sub-image-based position. In some examples, the identifier may be a syntax element subPicIdx. For example, multiple sub-image-based identifiers may be included in the PPS. When extracting a sub-image, the corresponding sub-image ID may be indicated in the PPS by using a flag / pointer and / or removing unused sub-image IDs. In some examples, an explicit ID flag may also be encoded in a parameter set. The flag may indicate to a decoder that a mapping relationship may be used to map the slice address from the picture-based position to the sub-picture-based position. In some examples, the mapping relationship may be a syntax element SliceSubpicToPicIdx[SubPicIdx][slice_address]. Accordingly, the flag may indicate that the slice address is not an index. In some examples, the flag may be sps_subpic_id_present_flag.
[0131] In step 907, a sub-code stream is extracted from the code stream. For example, extracting the sub-code stream from the code stream may include: extracting the first slice without rewriting the slice header according to the slice address of the first slice, the slice address length and the identifier. In a specific example, such extraction may also include extracting a sub-image from the image. In this case, the sub-image includes the first slice. The parameter set may also be included in the sub-code stream. For example, the sub-code stream may include the sub-image, the slice header, the PPS, the SPS, etc.
[0132] In step 909, the sub-stream is stored to be sent to a decoder. The sub-stream can then be sent to the decoder as needed.
[0133] Fig.10The flowchart of an exemplary method 1000 for decoding a sub-code stream (e.g., sub-code stream 501) including a sub-image (e.g., sub-image 522), wherein the sub-code stream including the sub-image is extracted from a code stream (e.g., code stream 500) of an image (e.g., image 600) through explicit address indication. The method 1000 may be performed by a decoder such as the encoding and decoding system 200, the decoder 400, and / or the video decoding device 800 when executing the method 100.
[0134] Method 1000 may begin when a decoder begins receiving a substream extracted from a codestream, such as the result of method 900. In step 1001, the substream is received. The substream includes a sub-image in an image. For example, a codestream encoded at an encoder side may include an image, and the encoder and / or striper extracts the substream from the codestream, the substream including a sub-image, the sub-image including one or more regions of the image in the codestream. The received sub-image may be divided into a plurality of stripes. The plurality of stripes may include a stripe designated as a first stripe. The first stripe may be any stripe in the image, but is described as the first stripe for clarity of discussion. For example, the upper left corner of the first stripe may not be aligned with the upper left corner of the image. The substream also includes a PPS, which describes syntax associated with the image, and therefore also describes syntax associated with the sub-image. The substream also includes a slice header, which describes syntax associated with the first stripe.
[0135] In step 1003, a parameter set such as a PPS and / or an SPS is parsed to obtain an explicit ID flag. The ID flag may indicate that a mapping relationship may be used to map a slice address from a position based on an image to a position based on a sub-image. Accordingly, the flag may indicate that the corresponding slice address includes a defined value and does not include an index. In some examples, the flag may be sps_subpic_id_present_flag. According to the value of the ID flag, the PPS may be parsed to obtain an identifier and a slice address length of the first slice. The identifier may be a sub-image identifier. The slice address length may indicate the number of bits included in the corresponding slice address. For example, the slice address length in the PPS may include data sufficient to parse the slice address from the slice header. In some examples, the length may be a syntax element subpic_id_len_minus1. In addition, the identifier may include data sufficient to map the slice address from a position based on an image to a position based on a sub-image. In some examples, the identifier may be a syntax element subPicIdx. For example, multiple sub-image-based identifiers may be included in the PPS. When extracting a sub-image, the corresponding sub-image ID may be indicated in the PPS by using a flag / pointer and / or removing unused sub-image IDs.
[0136] In step 1005, a slice address of the first slice is determined from the slice header according to the identifier and the slice address length. For example, the length in the PPS can be used to determine the bit boundary to parse the slice address from the slice header. The identifier and the slice address can then be used to map the slice address from a picture-based position to a sub-picture-based position. For example, the mapping relationship between the picture-based position and the sub-picture-based position can be used to align the slice header with the sub-picture. In this way, the decoder can solve the address mismatch problem between the slice header and the picture addressing scheme caused by the encoder and / or slicer performing sub-stream extraction without rewriting the slice header. In some examples, the mapping relationship can be a syntax element SliceSubpicToPicIdx[SubPicIdx][slice_address].
[0137] In step 1007, the sub-stream is decoded to generate a video sequence of sub-images. The sub-images may include the first stripe. Accordingly, the first stripe is also decoded. Then, the video sequence including the sub-images (including the decoded first stripe) may be sent to be displayed through a head mounted display or other display device.
[0138] Fig.11A schematic diagram of an exemplary system 1100 for transmitting a sub-stream (e.g., sub-stream 501) including a sub-image (e.g., sub-image 522). The sub-stream including the sub-image is extracted from a stream (e.g., stream 500) of an image (e.g., image 600) through explicit address indication. The system 1100 can be implemented by an encoder and a decoder such as the encoding and decoding system 200, the encoder 300, the decoder 400, and / or the video decoding device 800. In addition, the system 1100 can be used to implement the method 100, the method 900, and / or the method 1000.
[0139] The system 1100 includes a video encoder 1102. The video encoder 1102 includes an encoding module 1101, which is used to: encode an image in a code stream, wherein the image includes multiple slices, and the multiple slices include a first slice; encode a slice header in the code stream, wherein the slice header includes a slice address of the first slice; and encode a PPS in the code stream, wherein the PPS includes an identifier and a slice address length of the first slice. The video encoder 1102 also includes an extraction module 1103, which is used to extract a sub-code stream in the code stream according to the slice address of the first slice, the slice address length and the identifier without rewriting the slice header. The video encoder 1102 also includes a storage module 1105, which is used to store the code stream to send it to a decoder. The video encoder 1102 also includes a sending module 1107, which is used to send the sub-code stream to the decoder, wherein the sub-code stream includes the slice header, the PPS, the first slice and / or the corresponding sub-image. The video encoder 1102 may also be used to perform any step of the method 900 .
[0140] The system 1100 further includes a video decoder 1110. The video decoder 1110 includes a receiving module 1111 for receiving a sub-stream, wherein the sub-stream includes a sub-image in an image, wherein the image is divided into a plurality of slices including a first slice, a PPS associated with the image and the sub-image, and a slice header associated with the first slice. The video decoder 1110 further includes a parsing module 1113 for parsing the PPS to obtain an identifier and a slice address length of the first slice. The video decoder 1110 further includes a determining module 1115 for determining a slice address of the first slice from the slice header according to the identifier and the slice address length. The video decoder 1110 further includes a decoding module 1117 for decoding the sub-stream to generate a video sequence including the sub-image of the first slice. The video decoder 1110 further includes a sending module 1119 for sending the video sequence including the sub-image of the first slice for display. The video decoder 1110 can also be used to perform any step of the method 1000.
[0141] A first component is directly coupled to a second component when there is no intermediate component between the first component and the second component other than a line, trace, or other medium. A first component is indirectly coupled to a second component when there is an intermediate component between the first component and the second component other than a line, trace, or other medium. The term "coupled" and variations thereof include direct coupling and indirect coupling. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the subsequent number.
[0142] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and some steps may be omitted or combined.
[0143] Although the present invention provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in a variety of other specific forms without departing from the spirit or scope of the present invention. The present examples should be considered illustrative rather than restrictive, and the present invention is not limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.
[0144] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, components, techniques or methods. Other examples of changes, substitutions and modifications can be determined by those skilled in the art and may be given without departing from the spirit and scope disclosed herein.
Claims
1. A method implemented in a decoder, characterized in that: The method comprises: A receiver of the decoder receives a code stream, wherein the code stream includes a sub-image in an image, wherein the image is divided into a plurality of slices including a first slice, a parameter set associated with the image and the sub-image, and a slice header associated with the first slice; The processor of the decoder parses the bitstream to obtain a first parameter in the parameter set, the first parameter being used to derive a stripe address length of the first stripe, the stripe address length being inferred to be equal to Ceil( Log2( NumTilesInPic )) according to whether the first parameter exists, wherein NumTilesInPic represents the number of tiles in the image; The processor determines, from the stripe header, a stripe address of the first stripe according to the stripe address length; The processor decodes the sub-image including the first slice according to the slice address.
2. The method according to claim 1, characterized in that The method further comprises: The processor sends a video sequence including the sub-image for display.
3. The method according to claim 1 or 2, characterized in that: The stripe address length indicates the number of bits included in the stripe address.
4. The method according to claim 1 or 2, characterized in that: The method further comprises: A processor of the decoder parses the parameter set to obtain an identifier.
5. The method according to claim 4, characterized in that Determining the stripe address of the first stripe includes: The processor uses the length in the parameter set to determine a bit boundary to parse the slice address from the slice header; The processor maps the slice address from a picture-based location to a sub-picture-based location using the identifier and the slice address.
6. The method according to claim 5, characterized in that The method further includes: the processor parsing the parameter set to obtain an identifier (ID) tag, wherein the ID tag indicates that a mapping relationship can be used to map the stripe address from the image-based position to a sub-image-based position.
7. The method according to claim 5 or 6, characterized in that: The mapping relationship between the image-based position and the sub-image-based position aligns the slice header with the sub-image without rewriting the slice header.
8. The method according to claim 1 or 2, characterized in that: The stripe address includes the definition value but not the index.
9. A video decoding device, characterized in that: The video decoding device comprises: A processor, a memory, a receiver coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the memory, the receiver and the transmitter are configured to execute the method according to any one of claims 1 to 8.
10. A non-transitory computer readable medium, characterized in that The non-transitory computer-readable medium includes a computer program product for use by a video decoding device; the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium; when a processor executes the computer-executable instructions, the video decoding device performs the method according to any one of claims 1 to 8.
