Arbitrary and wrap-around block grouping

Abstract

The present invention discloses a video decoding mechanism. The mechanism includes dividing a first image into a plurality of blocks. A group of the blocks are allocated to a first block group. The first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction. The first block group is encoded into a code stream. A flag is encoded into a parameter set in the code stream to indicate that the first block group is an arbitrary block group. The code stream is stored for sending to a decoder.

Description

[0001] Cross-application of related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 780,053, filed on December 14, 2018 by FNU Hendry et al., entitled “Arbitrary and Wrap-Around Tile Grouping,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to video coding and, more particularly, to partition groups that wrap around picture boundaries to increase compression in video coding. Background Art

[0004] Even in the case of shorter videos, a large amount of video data needs to be described, which can cause difficulties when the data is to be sent or otherwise transmitted over a communication network with limited bandwidth capacity. Therefore, video data is usually compressed before being sent over modern telecommunications networks. Due to limited memory resources, the size of the video can also become an issue when storing the video in a storage device. Video compression devices usually use software and / or hardware on the source side to decode the video data before sending or storing, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received on the destination side by a video decompression device for decoding the video data. With limited network resources and a growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the invention

[0005] In one embodiment, the present invention includes a method implemented in an encoder, the method comprising: a processor of the encoder divides a first image into a plurality of blocks; the processor assigns a group of the blocks to a first block group, wherein the first block group is a rectangular shape surrounding the first image in a horizontal direction and a vertical direction; the processor encodes the first block group into a code stream; the processor encodes a flag into a parameter set in the code stream to indicate that the first block group is an arbitrary block group; and the code stream is stored in a memory of the encoder, the code stream being sent to a decoder. Some systems use block groups that surround an image in a horizontal direction or a vertical direction, but not both. Such a system may determine the number of blocks in a block group by subtracting the index of the first block of the block group from the index of the last block of the block group. Such a system may not be able to correctly determine the number of blocks in a block group if the image is surrounded in both directions. The present aspect provides a mechanism to correctly determine the correct number of blocks in a block group regardless of how the block group surrounds the image, including when it is surrounded in both directions.

[0006] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the flag is arbitrary_tile_group_present_flag.

[0007] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the flag is encoded into a picture parameter set (PPS) associated with the first image.

[0008] Optionally, according to any of the above aspects, in another implementation of the aspect, the processor is further included encoding a first block index and a last block index into a block group header, wherein the first block index represents the first block in the first block group, and the last block index represents the last block in the first block group.

[0009] Optionally, according to any of the above aspects, in another implementation of the aspect, it also includes determining the number of blocks in the second block group segmented from the second image by the following steps: setting the incremental block index to the difference between the last block index of the second block group and the first block index of the second block group; determining the number of block rows in the second block group by dividing the incremental block index by the number of block columns in the second image plus one; determining the number of block columns in the second block group as the incremental block index modulo the number of block columns in the second image plus one. This aspect describes an exemplary mechanism for determining the number of columns and rows in a block group based only on the last block index and the first block index. This aspect addresses the situation where no block wrapping occurs.

[0010] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is less than the first block index of the second block group (indicating vertical wrapping), and when the first block column index is greater than the last block column index (indicating horizontal wrapping), the incremental block index increases the number of blocks in the second image and the number of block columns in the second image. This aspect modifies the above aspect so as to perform correct calculation when both horizontal wrapping and vertical wrapping exist.

[0011] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the tile rows in the second tile group and the number of the tile columns in the second tile group, when the last tile index of the second tile group is less than the first tile index of the second tile group (indicating vertical wrapping), and when the first tile column index is not greater than the last tile column index (indicating no horizontal wrapping), the incremental tile index increases the number of tiles in the second image. This aspect modifies the above aspect so that correct calculation is performed when there is only vertical wrapping.

[0012] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is not less than the first block index of the second block group (indicating no vertical wrapping), and when the first block column index is greater than the last block column index (indicating horizontal wrapping), the incremental block index increases the number of the block columns in the second image. This aspect modifies the above aspect so that correct calculation is performed when there is only horizontal wrapping.

[0013] In one embodiment, the present invention includes a method implemented in a decoder, the method including: a processor of the decoder receives a code stream through a receiver, the code stream including a parameter set and a first block group, the first block group including a group of blocks segmented from a first image; the processor obtains a flag from the parameter set, wherein the flag indicates that the first block group is an arbitrary block group; the processor determines the number of blocks in the first block group based on the flag, a first block index, and a last block index, wherein the first block group is a rectangular shape surrounding the first image in a horizontal direction and a vertical direction; the processor decodes the first block group to generate a reconstructed video sequence for display. Some systems use block groups that surround an image in either a horizontal direction or a vertical direction, but not both. Such a system may determine the number of blocks in the block group by subtracting the index of the first block of the block group from the index of the last block of the block group. If the image is surrounded in both directions, such a system may not be able to correctly determine the number of blocks in the block group. The present aspect provides a mechanism to correctly determine the correct number of tiles in a tile group no matter how the tile group wraps around an image, including when wrapping in both directions.

[0014] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the flag is arbitrary_tile_group_present_flag.

[0015] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the marker is obtained from a PPS associated with the first image.

[0016] Optionally, according to any of the above aspects, in another implementation of the aspect, the processor is also included in obtaining the first block index and the last block index from the block group header, wherein the first block index represents the first block in the first block group, and the last block index represents the last block in the first block group.

[0017] Optionally, according to any of the above aspects, in another implementation of the aspect, it also includes determining the number of blocks in the second block group segmented from the second image by the following steps: setting the incremental block index to the difference between the last block index of the second block group and the first block index of the second block group; determining the number of block rows in the second block group by dividing the incremental block index by the number of block columns in the second image plus one; determining the number of block columns in the second block group as the incremental block index modulo the number of block columns in the second image plus one. This aspect describes an exemplary mechanism for determining the number of columns and rows in a block group based only on the last block index and the first block index. This aspect addresses the situation where no block wrapping occurs.

[0018] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is less than the first block index of the second block group (indicating vertical wrapping), and when the first block column index is greater than the last block column index (indicating horizontal wrapping), the incremental block index increases the number of blocks in the second image and the number of block columns in the second image. This aspect modifies the above aspect so as to perform correct calculation when both horizontal wrapping and vertical wrapping exist.

[0019] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the tile rows in the second tile group and the number of the tile columns in the second tile group, when the last tile index of the second tile group is less than the first tile index (indicating vertical wrapping), and when the first tile column index is not greater than the last tile column index, indicating no horizontal wrapping, the incremental tile index increases the number of tiles in the second image. This aspect modifies the above aspect so that correct calculation is performed when there is only vertical wrapping.

[0020] Optionally, according to any of the above aspects, in another implementation of the aspect, before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is not less than the first block index (indicating no vertical wrapping), and when the first block column index is greater than the last block column index (indicating horizontal wrapping), the incremental block index increases the number of the block columns in the second image. This aspect modifies the above aspect so that correct calculation is performed when there is only horizontal wrapping.

[0021] In one embodiment, the present invention includes a video decoding device, including: a processor, a receiver coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, and the transmitter are configured to execute the method according to any one of the above aspects.

[0022] In one embodiment, the present invention includes a non-transitory computer-readable medium, which includes a computer program product for use by a video decoding device, and the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium. When a processor executes the computer-executable instructions, the video decoding device performs the method described in any one of the above aspects.

[0023] In one embodiment, the present invention includes an encoder, comprising: a segmentation module, used to segment a first image into multiple blocks; an allocation module, used to allocate a group of the blocks to a first block group, wherein the first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction; an encoding module, used to: encode the first block group into a code stream; encode a flag into a parameter set in the code stream to indicate that the first block group is an arbitrary block group; and a storage module, used to store the code stream for sending to a decoder.

[0024] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the encoder is also used to execute the method according to any of the above aspects.

[0025] In one embodiment, the present invention includes a decoder, the decoder including: a receiving module, used to receive a code stream, the code stream including a first block group, the first block group including a group of blocks segmented from a first image; an acquisition module, used to obtain a flag from a parameter set in the code stream, wherein the flag indicates that the first block group is an arbitrary block group; a determination module, used to determine the number of blocks in the first block group based on the flag, a first block index and a last block index, wherein the first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction; a decoding module, used to decode the first block group to generate a reconstructed video sequence for display.

[0026] Optionally, according to any of the above aspects, in another implementation manner of the aspect, the decoder is further used to execute the method according to any of the above aspects.

[0027] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0028] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] For a more complete understanding of the present invention, reference is made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0030] Figure 1 A flow chart of an exemplary method of decoding a video signal.

[0031] Figure 2 is a schematic diagram of an exemplary codec system for video decoding.

[0032] Figure 3 is a schematic diagram of an exemplary video encoder.

[0033] Figure 4 is a schematic diagram of an exemplary video decoder.

[0034] Figure 5 is a schematic diagram of an exemplary code stream including an encoded video sequence.

[0035] Figures 6A-6D An exemplary image segmented into exemplary groups of tiles is shown.

[0036] Figure 7 is a schematic diagram of an exemplary video decoding device.

[0037] Figure 8 A flowchart of an exemplary method for encoding an image with arbitrary block groups into a codestream.

[0038] Fig. 9 A flowchart of an exemplary method for decoding an image with arbitrary partition groups from a code stream.

[0039] Fig.10 A flow chart of an exemplary method for determining a partition group configuration from an arbitrary partition group based on a first partition index and a last partition index.

[0040] Fig.11 A schematic diagram of an exemplary system for decoding a video sequence of pictures in a codestream. DETAILED DESCRIPTION

[0041] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. The present invention should not be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and the full scope of their equivalents.

[0042] This document uses various abbreviations, such as coding tree block (CTB), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), Joint Video Experts Team (JVET), motion constrained tile set (MCTS), maximum transfer unit (MTU), network abstraction layer (NAL), picture order count (POC), raw byte sequence payload (RBSP), sequence parameter set (SPS), versatile video coding (VVC), and working draft (WD).

[0043] Many video compression techniques can be used to reduce the size of video files while minimizing data loss. For example, video compression techniques can include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in a video sequence. For block-based video decoding, a video slice (e.g., a video image or a portion of a video image) can be divided into video blocks, which can also be referred to as tree blocks, coding tree blocks (CTB), coding tree units (CTU), coding units (CU) and / or coding nodes. Video blocks in an intra-frame decoding (I) slice of an image are decoded using spatial prediction with reference to reference samples in neighboring blocks in the same image. Video blocks in an inter-frame decoding unidirectional prediction (P) or bidirectional prediction (B) slice of an image can be decoded using spatial prediction with reference to reference samples in neighboring blocks in the same image, or can be decoded using temporal prediction with reference to reference samples in other reference images. A picture (picture / image) can be referred to as a frame, and a reference picture can be referred to as a reference frame. Spatial or temporal prediction produces a prediction block representing an image block. The residual data represents the pixel difference between the original image block and the prediction block. Therefore, the inter-frame decoded block is encoded based on the motion vector pointing to the reference sample block constituting the prediction block, and the residual data representing the difference between the decoded block and the prediction block. The intra-frame decoded block is encoded according to the intra-frame decoding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to produce residual transform coefficients, which can be quantized. The quantized transform coefficients are initially arranged in a two-dimensional array. The quantized transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients. Entropy coding can be used to achieve greater compression. This video compression technique is described in detail below.

[0044] In order to ensure that the encoded video can be accurately decoded, the video is encoded and decoded according to the corresponding video coding standards. Video coding standards include international telecommunication union (ITU) standardization sector (ITU-T) H.261, international organization for standardization (ISO) / international electrotechnical commission (IEC) motion picture experts group (MPEG)-1 part 2, ITU-T H.262 or ISO / IEC MPEG-2 part 2, ITU-T H.263, ISO / IEC MPEG-4 part 2, advanced video coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 part 10), and high efficiency video coding (HEVC) (also known as ITU-T H.265 or MPEG-H part 2). AVC includes scalable video coding (SVC), multiview video coding (MVC), and multiview video coding plus depth (MVC+D), as well as extensions such as three dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as scalable HEVC (SHVC), multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The joint video experts team (JVET) of ITU-T and ISO / IEC has begun developing a video coding standard called versatile video coding (VVC). VVC is included in a working draft (WD), which includes JVET-L1001-v9.

[0045] In order to encode a video image, the image is first segmented and the resulting parts are encoded into a bitstream. There are various image segmentation schemes. For example, the image can be segmented into regular strips, non-independent strips, tiles, and / or the image can be segmented according to wavefront parallel processing (WPP). For simplicity, HEVC restricts the encoder so that only regular strips, non-independent strips, tiles, WPP and their combinations can be used when segmenting strips into CTB groups for video decoding. This segmentation can be used to support maximum transfer unit (MTU) size matching, parallel processing and reduce end-to-end delay. MTU represents the maximum amount of data that can be sent in a single message. If the message payload exceeds the MTU, the payload is divided into two messages through a process called segmentation.

[0046] A regular slice, also referred to as a slice, is a segmented portion of an image that can be reconstructed independently of other regular slices in the same image, although there are some interdependencies due to loop filtering operations. Each regular slice is encapsulated in its own network abstraction layer (NAL) unit for transmission. In addition, intra prediction (intra-frame sample prediction, motion information prediction, decoding mode prediction) and entropy coding dependencies across slice boundaries can be disabled to support independent reconstruction. This independent reconstruction supports parallelization operations. For example, parallelization based on regular slices reduces inter-processor or inter-core communication. However, since each regular slice is independent, each slice is associated with a separate slice header. Due to the bit cost of the slice header of each slice and the lack of prediction across slice boundaries, the use of regular slices will cause large decoding overhead. In addition, regular slices can be used to support MTU size matching requirements. Specifically, since regular slices are encapsulated in separate NAL units and can be decoded independently, each regular slice should be smaller than the MTU in the MTU scheme to avoid dividing the slice into multiple messages. Therefore, the goals of parallelization and the goal of MTU size matching may place conflicting requirements on the layout of stripes in an image.

[0047] Dependent slices are similar to normal slices, but have shortened slice headers and allow for splitting of picture tree block boundaries without affecting intra prediction. Therefore, dependent slices split normal slices into multiple NAL units, which reduces end-to-end latency by sending part of a normal slice before the encoding of the entire normal slice is completed.

[0048] A block is a partition of an image divided by horizontal and vertical boundaries, which form columns and rows of blocks. Blocks can be decoded in raster scan order (from right to left, from top to bottom). The scanning order of CTB is performed within the block. Therefore, the CTB in the first block is decoded in raster scan order, and then proceeds to the CTB in the next block. Similar to conventional slices, blocks eliminate intra-frame prediction dependencies and entropy decoding dependencies. However, blocks may not be included in each NAL unit, so blocks may not be used for MTU size matching. Each block can be processed by one processor / core, and the inter-processor / inter-core communication for intra-frame prediction between processing units for decoding adjacent blocks can be limited to sending shared slice headers (when adjacent blocks are in the same slice), and sharing reconstructed samples and metadata related to loop filtering. When multiple blocks are included in a slice, in addition to the first entry point offset in the slice, the entry point byte offset of each block can also be indicated (signaled) in the slice header. For each slice and partition, at least one of the following conditions must be met: (1) all coding tree blocks in a slice belong to the same partition; (2) all coding tree blocks in a partition belong to the same slice.

[0049] In WPP, the image is partitioned into a single row of CTBs. Entropy decoding and prediction mechanisms can use data from CTBs in other rows. Parallel processing is supported by decoding CTB rows in parallel. For example, the current row can be decoded in parallel with the previous row. However, the decoding of the current row is delayed by two CTBs from the decoding process of the previous rows. This delay ensures that data related to the CTBs above and to the right of the current CTB in the current row are available before the current CTB is decoded. When represented graphically, this approach appears as a wavefront. This staggered start can be parallelized using as many processors / cores as there are CTB rows included in the image. Since intra-frame prediction can be performed between adjacent treeblock rows within the image, it is important to use inter-processor / inter-core communication to implement intra-frame prediction. WPP partitioning does not take into account the NAL unit size. Therefore, WPP does not support MTU size matching. However, regular slices can be used in conjunction with WPP (which requires some decoding overhead) to achieve MTU size matching as needed.

[0050] Tiles may also include motion constrained tilesets. A motion constrained tileset (MCTS) is a set of tiles that constrains the associated motion vectors to point to integer sample positions and fractional sample positions within the MCTS, with fractional sample positions requiring only integer sample positions within the MCTS for interpolation. In addition, motion vector candidates are not allowed to be used for temporal motion vector prediction derived from blocks outside the MCTS. In this way, each MCTS can be decoded independently without including tiles in the MCTS. Temporal MCTS supplemental enhancement information (SEI) messages can be used to indicate the presence of MCTS in a bitstream and indicate the MCTS. The MCTS SEI message provides supplementary information that can be used for MCTS substream extraction (expressed as part of the SEI message semantics) to generate a bitstream that conforms to the MCTS. The information includes multiple extraction information sets, each of which defines multiple MCTSs, and includes raw bytes sequence payload (RBSP) bytes of replacement video parameter sets (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) used in the MCTS substream extraction process. When extracting a substream according to the MCTS substream extraction process, the parameter sets (VPS, SPS, PPS) can be rewritten or replaced, and the slice header can be updated because one or all slice address related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) can use different values ​​in the extracted substream.

[0051] When segmenting an image for further encoding, various block segmentation schemes can be used. As a specific example, blocks can be assigned to block groups, which in some examples can replace strips. In some examples, each block group can be extracted independently of other block groups. Therefore, block grouping can support parallelization by assigning each block group to a different processor. Block grouping can also be used in situations where it is not desired for the decoder to decode the entire image. As a specific example, a video decoding scheme can be used to support virtual reality (VR) video.

[0052] In VR video, one or more cameras can record the environment around the cameras. The user can then watch the VR video as if they were at the location of the cameras. In VR video, the image includes the entire environment around the user. The user then watches sub-portions of the image. For example, the user can use a head-mounted display that changes the sub-portion of the displayed image based on the user's head movements. The portion of the video that is displayed can be called a viewing angle.

[0053] Tiled grouping can be used to support more efficient video decoding of VR videos. For example, VR videos can be segmented into tile groups and encoded at multiple resolutions. The decoder can indicate the user's current perspective during streaming. The video server providing the VR video data can then forward the tile groups associated with the perspective at high resolution and forward the non-viewed tile groups at low resolution. In this way, the user can watch high-resolution VR video without sending the entire image at high resolution. The non-viewed sub-portions are discarded, so the user does not perceive the low resolution. However, if the user changes the perspective, the low-resolution tile group can be displayed to the user. Then, as the video plays, the resolution of the new perspective can be increased.

[0054] A tile group can divide an image into rectangles. The tiles included in the tile group can be indicated by indicating the first tile and the last tile of the tile group. In this case, the tile index of the first tile can be less than the tile index of the last tile. The decoder can calculate the configuration of the tile group by comparing these values. This method can efficiently indicate the tile group information. However, in some cases, the encoder may determine to wrap the tile group on the image boundary. When the top row of the tile group is toward the bottom of the image and the bottom row of the tile group is toward the top of the image, the tile group wraps around the image in the vertical direction. When the left column of the tile group is toward the right side of the image and the right column of the tile group is toward the left side of the image, the tile group wraps around the image in the horizontal direction. Formally speaking, horizontal wrapping occurs when the left image boundary and the right image boundary bisect a tile group, and vertical wrapping occurs when the upper image boundary and the lower image boundary bisect a tile group. This tile wrapping may occur particularly in VR video. This is because the VR image can include the entire spherical environment, flattened into a rectangular image for encoding. Therefore, picture boundaries in VR images may have a certain degree of arbitrariness, and in some cases, wrapping of tile groups across these arbitrary picture boundaries may result in increased compression. Such wrapping tile groups may have a first tile with a tile index greater than the tile index of the last tile, which may cause the decoder to miscalculate the configuration of the tile group.

[0055] Various mechanisms are disclosed herein for supporting accurate indication of tile groups when these tile groups wrap across image boundaries in the horizontal direction, vertical direction, and / or both. The encoder may encode arbitrary_tile_group_present_flag into a parameter set such as a picture parameter set (PPS) to indicate the presence of such tile grouping in the image. The encoder may also encode the first tile index and the last tile index of the tile group, for example, in a tile group header. The decoder may determine an incremental tile index, which is the difference between the last tile index and the first tile index. The decoder may determine the nature of the tile group wrap (if any) and adjust the incremental tile index accordingly. If the last tile index of the tile group is less than the first tile index of the tile group, the tile group includes vertical wrapping. If the first tile column index is greater than the last tile column index, the tile group includes horizontal wrapping. If the tile group includes both vertical wrapping and horizontal wrapping, the incremental tile index increases the number of tiles in the image and the number of tile columns in the image. If the block group includes vertical wrapping and no horizontal wrapping, the incremental block index increases the number of blocks in the image. If the block group includes horizontal wrapping and no vertical wrapping, the incremental block index increases the number of block columns in the image. Then, the decoder can use the incremental block index to determine the number of block rows and the number of block columns in the block group. For example, the number of block rows in the block group can be determined as the incremental block index divided by the number of block columns in the image plus one. In addition, the number of block columns in the block group can be determined as the incremental block index modulo the number of block columns in the image plus one.

[0056] Figure 1 Flowchart of an exemplary method 100 of operation for decoding a video signal. Specifically, the video signal is encoded at the encoder side. The encoding process compresses the video signal using various mechanisms to reduce the video file size. The smaller file size is advantageous for sending the compressed video file to the user while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally corresponds to the encoding process so that the decoder can consistently reconstruct the video signal.

[0057] In step 101, a video signal is input to an encoder. For example, the video signal can be an uncompressed video file stored in a memory. In another example, the video file can be captured by a video capture device (e.g., a video camera) and encoded to support live broadcasting of the video. The video file can include an audio component and a video component. The video component includes a series of image frames that, when viewed in sequence, produce a visual effect of motion. These frames include pixels represented by light (referred to herein as luminance components (or luminance samples)) and color (referred to as chrominance components (or chrominance samples)). In some examples, these frames can also include depth values ​​to support three-dimensional viewing.

[0058] In step 103, the video is segmented into blocks. Segmentation includes subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC) (also known as H.265 and MPEG-H Part 2), the frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64×64 pixels). The CTU includes luma samples and chroma samples. The coding tree can be used to divide the CTU into blocks, and then recursively subdivide these blocks until a configuration that supports further encoding is obtained. For example, the luma component of the frame can be subdivided until each block includes a relatively uniform luma value. In addition, the chroma component of the frame can be subdivided until each block includes a relatively uniform color value. Therefore, the segmentation mechanism varies depending on the content of the video frame.

[0059] In step 105, various compression mechanisms are used to compress the image blocks segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction is intended to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Therefore, there is no need to repeatedly describe the blocks depicting the object in the reference frame in adjacent frames. Specifically, an object (such as a table) can remain in a constant position in multiple frames. Therefore, the table is only described once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects in multiple frames. In addition, due to object movement or camera movement, etc., a moving object can be represented by multiple frames. As a specific example, a video can show a car moving on the screen through multiple frames. Motion vectors can be used to describe this movement. A motion vector is a two-dimensional vector that provides an offset from the coordinates of an object in a frame to the coordinates of the object in a reference frame. Therefore, inter-frame prediction can encode the image blocks in the current frame as a set of motion vectors, representing the offset relative to the corresponding blocks in the reference frame.

[0060] Intra prediction encodes blocks in a common frame. Intra prediction exploits the fact that luminance and chrominance components tend to cluster in a frame. For example, a patch of green in part of a tree tends to be adjacent to several similar patches of green. Intra prediction uses a variety of directional prediction modes (e.g., 33 modes in HEVC), plane mode, and direct current (DC) mode. The directional mode indicates that the current block is similar / identical to the samples of the neighboring blocks in the corresponding direction. The plane mode indicates that a series of blocks on a row / column (e.g., a plane) can be interpolated based on the neighboring blocks at the edge of the row. In fact, the plane mode represents the smooth transition of light / color on the row / column by using a relatively constant slope of the changing value. The DC mode is used for boundary smoothing, indicating that the block is similar / identical to the average value associated with the samples of all neighboring blocks, which are associated with the angular direction of the directional prediction mode. Therefore, the intra prediction block can represent the image block as various relational prediction mode values ​​instead of the actual value. In addition, the inter prediction block can represent the image block as a motion vector value instead of the actual value. In either case, the prediction block may not accurately represent the image block in some cases. All difference values ​​are stored in the residual block. The residual block can be transformed to further compress the file.

[0061] In step 107, various filtering techniques can be applied. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above may create a blocky image on the decoder side. In addition, a block-based prediction scheme can encode a block and then reconstruct the coded block for later use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to a block / frame. These filters reduce such block artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference block, making it less likely that the artifact will produce other artifacts in subsequent blocks encoded according to the reconstructed reference block.

[0062] In step 109, once the video signal has been segmented, compressed and filtered, the resulting data is encoded into a bitstream. The bitstream includes the above data and any indicative data required to support appropriate reconstruction of the video signal at the decoder side. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in a memory so that it can be sent to the decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating a bitstream is an iterative process. Therefore, steps 101, 103, 105, 107 and 109 can be performed continuously and / or simultaneously in multiple frames and blocks. Figure 1The order shown is presented for clarity and ease of description and is not intended to limit the video coding process to a particular order.

[0063] In step 111, the decoder receives the code stream and starts the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the code stream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the code stream to determine the segmentation of the frame. The segmentation should match the block segmentation result in step 103. The entropy coding / entropy decoding used in step 111 is described here. The encoder makes many choices during the compression process, such as selecting a block segmentation scheme from multiple possible choices based on the spatial positioning of values ​​in one or more input images. Indicating the exact choice may use a large number of bits. The bits used in this article are binary values ​​that are considered variables (for example, bit values ​​that can vary depending on the context). Entropy coding allows the encoder to discard any options that are obviously not suitable for a particular situation, leaving a set of usable options. Then, a codeword is assigned to each usable option. The length of the codeword depends on the number of usable options (for example, one bit corresponds to two options, two bits correspond to three or four options, etc.). Then, the encoder encodes the codeword of the selected option. This scheme reduces the size of the codeword because the codeword is as large as needed to uniquely indicate a selection from a small subset of available options, rather than as large as needed to uniquely indicate a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of available options in a similar manner to the encoder. By determining the set of available options, the decoder can read the codeword and determine the selection made by the encoder.

[0064] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. Then, the decoder uses the residual block and the corresponding prediction block to reconstruct the image block according to the segmentation. The prediction block may include an intra-frame prediction block and an inter-frame prediction block generated by the encoder side in step 105. Then, the reconstructed image block is positioned in the frame of the reconstructed video signal according to the segmentation data determined in step 111. The syntax of step 113 can also be indicated in the bitstream by entropy coding described above.

[0065] In step 115, the frames of the reconstructed video signal are filtered at the encoder side in a manner similar to step 107. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter may be applied to the frames to remove blocking artifacts. Once the frames are filtered, in step 117, the video signal may be output to a display for viewing by an end user.

[0066] Figure 21 is a schematic diagram of an exemplary encoding and decoding (codec) system 200 for video decoding. Specifically, the codec system 200 can implement the operating method 100. The codec system 200 broadly describes the components used in an encoder and a decoder. As described in steps 101 and 103 of the operating method 100, the codec system 200 receives a video signal and segments the video signal to generate segmented video signals 201. As described in steps 105, 107 and 109 of the method 100, the codec system 200 then compresses the segmented video signals 201 into encoded bitstreams when acting as an encoder. As described in steps 111, 113, 115 and 117 of the operating method 100, the codec system 200 generates an output video signal from the bitstream when acting as a decoder. The codec system 200 includes an overall decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded image buffer component 223, and a header formatting and context adaptive binary arithmetic coding (context adaptive binary arithmetic coding, CABAC) component 231. These components are coupled as shown. Figure 2 In FIG. 2 , the black lines represent the movement of data to be encoded / decoded, while the dotted lines represent the movement of control data that controls the operation of other components. The components of the codec system 200 may all be present in the encoder. The decoder may include a subset of the components of the codec system 200. For example, the decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described here.

[0067] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. These blocks can be referred to as nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks may be included in a coding unit (coding unit, CU). For example, a CU may be a sub-part of a CTU, which includes a luminance block, a red difference chrominance (Cr) block, and a blue difference chrominance (Cb) block and corresponding syntax instructions for the CU. The partitioning mode may include a binary tree (BT), a triple tree (TT), and a quad tree (QT), which are used to partition a node into two, three, or four child nodes of different shapes, respectively, according to the partitioning mode used. The segmented video signal 201 is forwarded to the overall decoder control component 211, the transform scaling and quantization component 213, the intra-frame estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.

[0068] The overall decoder control component 211 is used to make decisions related to encoding the images of the video sequence into the bitstream according to the application constraints. For example, the overall decoder control component 211 manages the optimization of the bitrate / bitstream size relative to the reconstruction quality. Such decisions can be made based on the storage space / bandwidth availability and the image resolution request. The overall decoder control component 211 also manages the buffer utilization according to the transmission speed to alleviate the buffer under-load and overload problems. In order to solve these problems, the overall decoder control component 211 manages the segmentation, prediction and filtering performed by other components. For example, the overall decoder control component 211 can dynamically increase the compression complexity to increase the resolution and increase the bandwidth utilization, or reduce the compression complexity to reduce the resolution and bandwidth utilization. Therefore, the overall decoder control component 211 controls other components of the codec system 200 to balance the video signal reconstruction quality and bitrate issues. The overall decoder control component 211 creates control data that controls the operation of other components. The control data is also forwarded to the header formatting and CABAC component 231 to be encoded into the codestream, indicating parameters for decoding in the decoder.

[0069] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices of the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 can perform multiple decoding processes to select an appropriate decoding mode for each video data block, etc.

[0070] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, a motion vector can represent the displacement of a decoding object relative to a prediction block. A prediction block is a block that is found to be highly matched to a block to be decoded in terms of pixel differences. A prediction block may also be referred to as a reference block. Such pixel differences can be determined by the sum of absolute difference (SAD), the sum of square difference (SSD), or other difference metrics. HEVC uses several decoding objects, including CTU, coding tree block (CTB), and CU. For example, a CTU can be divided into multiple CTBs, which can then be divided into multiple CBs, and multiple CBs are used to be included in a CU. A CU can be encoded as a prediction unit (PU) including prediction data and / or a transform unit (TU) including transform residual data of a CU. The motion estimation component 221 generates motion vectors, PUs, and TUs using rate-distortion analysis as part of a rate-distortion optimization process. For example, the motion estimation component 221 can determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and can select reference blocks, motion vectors, etc. with optimal rate-distortion characteristics. The optimal rate-distortion characteristics balance the quality of video reconstruction (e.g., the amount of data lost due to compression) and decoding efficiency (e.g., the size of the final encoding).

[0071] In some examples, the codec system 200 can calculate the value of the sub-integer pixel position of the reference image stored in the decoded image buffer component 223. For example, the video codec system 200 can interpolate the values ​​of the quarter pixel position, the eighth pixel position, or other fractional pixel position of the reference image. Therefore, the motion estimation component 221 can perform a motion search relative to the full pixel position and the fractional pixel position, and output a motion vector with fractional pixel accuracy. The motion estimation component 221 calculates the motion vector of the PU of the video block in the inter-frame decoding slice by comparing the position of the PU with the position of the prediction block of the reference image. The motion estimation component 221 outputs the calculated motion vector as motion data to the header formatting and CABAC component 231 for encoding, and outputs the motion to the motion compensation component 219.

[0072] The motion compensation performed by the motion compensation component 219 may involve obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. In addition, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. After receiving the motion vector of the PU of the current video block, the motion compensation component 219 may locate the prediction block pointed to by the motion vector. Then, the pixel difference values ​​are formed by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the decoded current video block, thereby forming a residual video block. Typically, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for the chroma component and the luma component. The prediction block and the residual block are forwarded to the transform scaling and quantization component 213.

[0073] The segmented video signal 201 is also sent to the intra-frame estimation component 215 and the intra-frame prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-frame estimation component 215 and the intra-frame prediction component 217 can be highly integrated, but are shown separately for conceptual purposes. The intra-frame estimation component 215 and the intra-frame prediction component 217 perform intra-frame prediction on the current block relative to the block in the current frame, replacing the inter-frame prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra-frame estimation component 215 determines an intra-frame prediction mode for encoding the current block. In some examples, the intra-frame estimation component 215 selects an appropriate intra-frame prediction mode from a plurality of tested intra-frame prediction modes to encode the current block. The selected intra-frame prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0074] For example, the intra-frame estimation component 215 uses a rate-distortion analysis of various tested intra-frame prediction modes to calculate rate-distortion values ​​and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block that is encoded to generate the coded block, as well as the code rate (e.g., the number of bits) used to generate the coded block. The intra-frame estimation component 215 calculates a ratio based on the distortion and rate of the various coded blocks to determine which intra-frame prediction mode exhibits the best rate-distortion value for the block. In addition, the intra-frame estimation component 215 can be used to decode the depth block of the depth map using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0075] When implemented on an encoder, the intra prediction component 217 can generate a residual block from the prediction block according to the selected intra prediction mode determined by the intra estimation component 215, or when implemented on a decoder, read the residual block from the code stream. The residual block includes the difference in values ​​between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra estimation component 215 and the intra prediction component 217 can operate on the luminance component and the chrominance component.

[0076] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform to the residual block, thereby generating a video block including residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform may convert the residual information from a pixel value domain to a transform domain, such as a frequency domain. The transform scaling and quantization component 213 is also used to scale the transformed residual information according to frequency, etc. This scaling involves applying a scaling factor to the residual information so as to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all coefficients. The degree of quantization may be modified by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 may then scan the matrix including the quantized transform coefficients. The quantized transform coefficients are forwarded to the header formatting and CABAC component 231 for encoding into the codestream.

[0077] The scaling and inverse transform component 229 applies the inverse operation of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, inverse transform and / or inverse quantization to reconstruct the residual block in the pixel domain, for example, for later use as a reference block, which can become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can calculate the reference block by adding the residual block back to the corresponding prediction block for motion estimation of subsequent blocks / frames. Filters are used to reconstruct the reference block to reduce artifacts generated during scaling, quantization and transformation. Such artifacts may cause inaccurate predictions (and generate other artifacts) when predicting subsequent blocks.

[0078] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block of the scaling and inverse transform component 229 can be merged with the corresponding prediction block of the intra-frame prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block instead. As Figure 2 The filter control analysis component 227 and the in-loop filter component 225 are highly integrated with other components in the image processing unit and can be implemented together, but are shown separately for conceptual purposes. The filter used to reconstruct the reference block is used for a specific spatial area and includes multiple parameters to adjust the way such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference block to determine the location where such filters should be applied and sets the corresponding parameters. Such data is forwarded to the header formatting and CABAC component 231 for encoding as filter control data. The in-loop filter component 225 applies such filters according to the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters can be used in the spatial domain / pixel domain (e.g., for reconstructed pixel blocks) or the frequency domain according to the example.

[0079] When acting as an encoder, the filtered reconstructed image blocks, residual blocks and / or prediction blocks are stored in the decoded image buffer component 223 for later use in motion estimation, as described above. When acting as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and forwards them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks and / or reconstructed image blocks.

[0080] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to the decoder. Specifically, the header formatting and CABAC component 231 generates various headers to encode control data (such as overall control data and filter control data). In addition, prediction data (including intra-frame prediction) and motion data, as well as residual data in the form of quantized transform coefficient data are encoded in the bitstream. The final bitstream includes all the information required for the decoder to reconstruct the original segmented video signal 201. Such information may also include an intra-frame prediction mode index table (also called a codeword mapping table), a definition of the coding context of various blocks, a representation of the most likely intra-frame prediction mode, a representation of segmentation information, etc. Such data can be encoded using entropy coding. For example, the information may be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. After entropy coding, the encoded bitstream may be sent to another device (e.g., a video decoder) or archived for subsequent transmission or retrieval.

[0081] Figure 3 is a block diagram of an exemplary video encoder 300. The video encoder 300 may be used to implement the encoding function of the codec system 200 and / or to implement steps 101, 103, 105, 107, and / or 109 of the operating method 100. The encoder 300 segments the input video signal to generate segmented video signals 301, wherein the segmented video signals 301 are substantially similar to the segmented video signals 201. Then, the segmented video signals 301 are compressed by the components of the encoder 300 and encoded into a bitstream.

[0082] Specifically, the segmented video signal 301 is forwarded to the intra prediction component 317 for intra prediction. The intra prediction component 317 may be substantially similar to the intra estimation component 215 and the intra prediction component 217. The segmented video signal 301 is also forwarded to the motion compensation component 321 for inter prediction based on the reference block in the decoded image buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks of the intra prediction component 317 and the motion compensation component 321 are forwarded to the transform and quantization component 313 for transforming and quantizing the residual blocks. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual blocks and the corresponding prediction blocks (together with the relevant control data) are forwarded to the entropy coding component 331 for encoding into the bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231.

[0083] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 to be reconstructed into a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to the example, the in-loop filter in the in-loop filter component 325 is also used for the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. As described with respect to the in-loop filter component 225, the in-loop filter component 325 can include a plurality of filters. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block by the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.

[0084] Figure 4 is a block diagram of an exemplary video decoder 400. The video decoder 400 may be used to implement the decoding function of the codec system 200 and / or implement steps 111, 113, 115, and / or 117 of the operation method 100. For example, the decoder 400 receives a bitstream from the encoder 300, and generates a reconstructed output video signal according to the bitstream for display to an end user.

[0085] The code stream is received by entropy decoding component 433. Entropy decoding component 433 is used to implement entropy decoding schemes, such as CAVLC, CABAC, SBAC, PIPE decoding or other entropy decoding techniques. For example, entropy decoding component 433 can use header information to provide context to explain other data encoded as code words in the code stream. Decoded information includes any information required to decode the video signal, such as overall control data, filter control data, segmentation information, motion data, prediction data, and quantized transform coefficients in the residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 to reconstruct the residual block. The inverse transform and quantization component 429 can be similar to the inverse transform and quantization component 329.

[0086] The reconstructed residual block and / or prediction block is forwarded to the intra prediction component 417 to be reconstructed into an image block according to the intra prediction operation. The intra prediction component 417 can be similar to the intra estimation component 215 and the intra prediction component 217. Specifically, the intra prediction component 417 uses the prediction mode to locate the reference block in the frame, and uses the residual block for the result to reconstruct the intra prediction image block. The reconstructed intra prediction image block and / or residual block and the corresponding inter prediction data are forwarded to the decoded image buffer component 423 through the in-loop filter component 425. The decoded image buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block and / or prediction block, and such information is stored in the decoded image buffer component 423. The reconstructed image block of the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 can be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses the motion vector of the reference block to generate a prediction block, and uses the residual block for the result to reconstruct the image block. The resulting reconstructed block can also be forwarded to the decoded image buffer component 423 through the in-loop filter component 425. The decoded image buffer component 423 continues to store other reconstructed image blocks, which can be reconstructed into frames through segmentation information. Such frames can also be located in a sequence. The sequence is output to the display as a reconstructed output video signal.

[0087] Figure 5 is a schematic diagram of an exemplary bitstream 500 including an encoded video sequence. For example, the bitstream 500 may be generated by the codec system 200 and / or the encoder 300, and used for decoding by the codec system 200 and / or the decoder 400. In another example, the bitstream 500 may be generated by the encoder in step 109 of the method 100, and used by the decoder in step 111.

[0088] The code stream 500 includes a sequence parameter set (SPS) 510, a plurality of picture parameter sets (PPS) 512, a partition group header 514, and image data 520. The SPS 510 includes sequence data common to all images in the video sequence of the code stream 500. Such data may include image size, bit depth, decoding tool parameters, bit rate limit, etc. The PPS 512 includes one or more specific parameters of the corresponding image. Therefore, each image in the video sequence may refer to a PPS 512. The PPS 512 may indicate decoding tools, quantization parameters, offsets, image-specific decoding tool parameters (e.g., filter control), etc. that may be used for the partitions in the corresponding image. The partition group header 514 includes specific parameters for each partition group in the image. Therefore, each partition group in the video sequence may have a partition group header 514. The partition group header 514 may include partition group information, picture order count (POC), reference picture list, prediction weight, partition entry point, deblocking parameters, etc. It should be noted that some systems refer to the block group header 514 as a slice header, and use this information to describe a slice instead of a block group.

[0089] The image data 520 includes video data encoded according to inter-frame prediction and / or intra-frame prediction and corresponding transformed and quantized residual data. These image data 520 are classified according to the segmentation used to segment the image before encoding. For example, the image in the image data 520 is divided into blocks 523. The blocks 523 are further divided into coding tree units (coding tree units, CTUs). The CTUs are further divided into coding blocks according to the coding tree. Then, the coding blocks can be encoded / decoded according to the prediction mechanism. The image (image / picture) can include one or more blocks 523.

[0090] A block 523 is a segmented portion of an image created by horizontal and vertical boundaries. The block 523 can be decoded in a raster scan order, and in different examples, prediction based on other blocks 523 may or may not be allowed. Each block 523 has a unique block index 524 in the image. The block index 524 is a digital identifier selected according to the process and can be used to distinguish one block 523 from another block 523. For example, the block index 524 can increase in value in a raster scan order. The raster scan order is from left to right and from top to bottom. It should be noted that in some examples, a block identifier (ID) can also be assigned to the block 523. The block ID is an assigned identifier that can be used to distinguish one block 523 from another block. In some examples, the block ID can be used for calculation instead of using the block index 524 for calculation. In addition, in some examples, the block ID can be assigned the same value as the block index 524.

[0091] The tile index 524 can be indicated to indicate a tile group that includes tile 523. For example, tile 523 can be part of a tile group that surrounds an image boundary. Flags such as arbitrary_tile_group_present_flag can be set in the tile group header 514 to indicate that an arbitrary tile group is in use. An arbitrary tile group is a tile group that is not limited to a rectangular grouping of adjacent tiles 523. The first tile index and the last tile index can be indicated in the tile group header 514. In some examples, the first tile index and the last tile index are indicated by corresponding tile IDs. The decoder can then determine the configuration of the tile group based on the flag, the first tile index, and the last tile index. For example, as described below with respect to Figures 6A-6D Described in detail, decoder can determine the configuration of block group by using predefined algorithm.Encoder can also use predefined algorithm during rate distortion optimization process, because encoder uses the process similar to decoder, so that when selecting the coding method optimized, predict the decoding result at decoder place.By only indicating the first block index and the last block index, rather than indicating the complete member of block group, can save a lot of positions.This improves decoding efficiency, and reduces the memory resource and the network resource that encoder and decoder use.Therefore, predefined algorithm is provided to determine the block group configuration around block group according to the first block index and the last block index, for coding optimization provides other functions and flexibility, keeps this decoding efficiency simultaneously.

[0092] Figures 6A-6DExemplary images 601, 611, 621, and 631 are shown partitioned into exemplary partition groups 609, 619, 629, and 639, respectively. For example, images 601, 611, 621, and 631 may be encoded into codestream 500 by codec system 200, encoder 300, and / or decoded from codestream 500 by codec system 200, decoder 400. In addition, images 601, 611, 621, and 631 may be partitioned to support encoding and decoding according to method 100. Figures 6A-6D An exemplary arbitrary block group is shown. Therefore, a decoder reading a flag indicating an arbitrary block group should be able to determine which type of block groups 609, 619, 629 and 639 is used and determine the corresponding block group configuration based only on the first block index and the last block index (or the corresponding first block ID and the last block ID).

[0093] refer to Fig. 6A , image 601 is a single image 601 in a video sequence. Image 601 may be encoded in a bitstream by an encoder and decoded from the bitstream by a decoder. Image 601 may include a left border 601a, a right border 601b, an upper border 601c, and a lower border 601d, which are edges of image 601 included for clarity of discussion. Image 601 may be segmented into blocks 603.

[0094] Block 603 is a sub-portion of image 601. Block 603 can be similar to block 523 in essence. Block 603 can be rectangular or square. In addition, block 603 can be divided into block row 605 and block column 607. Block row 605 is a group of blocks 603, positioned in a horizontally adjacent manner to create a continuous line from left boundary 601a to right boundary 601b (and vice versa). Block column 607 is a group of blocks 603, positioned in a vertically adjacent manner to create a continuous line from upper boundary 601c to lower boundary 601d (and vice versa). Each block 603 is assigned a block index, and the block index increases in raster scan order. In the example shown, the block index is extended from 0 to 23. These block indexes are exemplary and are provided for clear discussion and should not be considered as limiting. Block 603 is included in one or more block groups 609.

[0095] The tile groups 609 are related sets of tiles 603 that can be extracted and decoded separately to support display of regions of interest and / or to support parallel processing, etc. The tiles 603 in the tile groups 609 can be decoded without reference to the tiles 603 outside the tile groups 609. Each tile 603 can be assigned to a corresponding tile group 609, so the image 601 can include multiple tile groups 609. However, for clarity of discussion, the present invention refers to the tile groups 609 shown as shaded areas, which include the tiles 603 indexed 7 to 10 and 13 to 16.

[0096] Therefore, the block group 609 of the image 601 can be indicated by the first block index 7 and the last block index 16. The decoder needs to determine the configuration of the block group 609 according to the first block index and the last block index. The configuration of the block group 609 used in this article represents the rows, columns and blocks 603 in the block group 609. In order to determine the configuration of the block group 609, the video decoding device can use a predefined algorithm. For example, the video encoding device determines the number of blocks 603 in the block group 609 segmented from the image 601 by setting the incremental block index to the difference between the last block index of the block group 609 and the first block index of the block group 609. The number of block rows 605 in the block group 609 can be determined by dividing the incremental block index by the number of block columns 607 in the image 601 plus 1. In addition, the number of block columns 607 in the block group 609 can be determined as the incremental block index modulo the number of block columns 607 in the image plus one. The number of tiles 603 in a tile group 609 may be determined by multiplying the number of tile columns 607 in the tile group 609 by the number of tile rows 605 in the tile group 609 .

[0097] Reference now Figure 6B, image 611 is substantially similar to image 601, with different block groups 619. Therefore, image 611 comprises left boundary 611a, right boundary 611b, upper boundary 611c, lower boundary 611d, block 613, block row 615 and block column 617, which are substantially similar to left boundary 601a, right boundary 601b, upper boundary 601c, lower boundary 601d, block 603, block row 605 and block column 607 respectively. Image 611 also comprises a block group 619 similar to block group 609. But block group 619 extends through upper boundary 611c and lower boundary 611d. Specifically, the first block index of block group 619 is 19, and the last block index is 4. Therefore, block group 619 is a rectangular shape that surrounds image 611 in the vertical direction. The video decoding device can determine the configuration of the block group 619 in substantially the same manner as the block group 609. However, the predefined algorithm can determine whether there is vertical surround and adjust the incremental block index to adjust the surround. For example, before determining the number of block rows 615 and the number of block columns 617 in the block group 619, the predefined algorithm can detect vertical surround. In a specific example, the predefined algorithm can determine that the last block index (e.g., 4) of the block group 619 is less than the first block index (e.g., 19) of the block group 619, which indicates vertical surround. The predefined algorithm can also determine that the first block column 617 index (e.g., 1) is not greater than the last block column 617 index (e.g., 4), which indicates that there is no horizontal surround. After determining that there is vertical surround and no horizontal surround, the incremental block index can increase the number of blocks 613 in the image 611 before determining the number of blocks 613, block rows 615, and block columns 617 in the block group 619.

[0098] Reference now Figure 6C, image 621 is substantially similar to image 601, has different block groups 629.Therefore, image 621 comprises left boundary 621a, right boundary 621b, upper boundary 621c, lower boundary 621d, block 623, block row 625 and block column 627, which are substantially similar to left boundary 601a, right boundary 601b, upper boundary 601c, lower boundary 601d, block 603, block row 605 and block column 607 respectively.Image 621 also comprises a block group 629 that is similar to block group 609.But block group 629 extends through left boundary 621a and right boundary 621b.Particularly, the first block index of block group 629 is 10, and the last block index is 13.Therefore, block group 629 is the rectangular shape that surrounds image 621 in the horizontal direction. The video decoding device can determine the configuration of the block group 629 in substantially the same manner as the block group 609. However, the predefined algorithm can determine whether there is horizontal surround and adjust the incremental block index to adjust the surround. For example, before determining the number of block rows 625 and the number of block columns 627 in the block group 629, the predefined algorithm can detect horizontal surround. In a specific example, the predefined algorithm can determine that the last block index (e.g., 13) of the block group 629 is not less than the first block index (e.g., 10) of the block group 629, which indicates that there is no vertical surround. The predefined algorithm can also determine that the first block column 627 index (e.g., 4) is greater than the last block column 627 index (e.g., 1), which indicates horizontal surround. After determining that there is horizontal surround and no vertical surround, before determining the number of blocks 623, block rows 625, and block columns 627 in the block group 629, the incremental block index can increase the number of block columns 627 in the image 621.

[0099] Reference now Fig.6D, image 631 is substantially similar to image 601, with different block groups 639. Therefore, image 631 includes left border 631a, right border 631b, upper border 631c, lower border 631d, block 633, block row 635 and block column 637, which are substantially similar to left border 601a, right border 601b, upper border 601c, lower border 601d, block 603, block row 605 and block column 607 respectively. Image 631 also includes a block group 639 similar to block group 609. However, block group 639 extends through left border 621a and right border 621b in the horizontal direction, and extends through upper border 631c and lower border 631d in the vertical direction. Specifically, the first block index of block group 639 is 16, and the last block index is 1. Therefore, block group 639 is a rectangular shape that surrounds image 631 in the horizontal direction and the vertical direction. The video decoding device can determine the configuration of the block group 639 in substantially the same manner as the block group 609. However, the predefined algorithm can determine whether there is horizontal surround and vertical surround, and adjust the incremental block index to adjust the surround. For example, before determining the number of block rows 635 and the number of block columns 637 in the block group 639, the predefined algorithm can detect horizontal surround and vertical surround. In a specific example, the predefined algorithm can determine that the last block index (e.g., 1) of the block group 639 is less than the first block index (e.g., 16) of the block group 639, which indicates vertical surround. The predefined algorithm can also determine that the first block column 637 index (e.g., 4) is greater than the last block column 637 index (e.g., 1), which indicates horizontal surround. After determining that there is vertical wrapping and horizontal wrapping, the incremental block index can increase the number of blocks 633 in the image 631 and the number of block columns 637 in the image 631 before determining the number of blocks 633, block rows 635 and block columns 637 in the block group 639.

[0100] As described above, the predefined algorithm may determine the tile group configuration without receiving data indicating the type of surround associated with the tile group. In a specific implementation, the following syntax may describe the predefined algorithm above.

[0101]

[0102]

[0103] The following are certain exemplary techniques for indicating the grouping of tiles and explicitly indicating tile IDs to facilitate extraction of tile groups from a codestream without having to modify the tile group header as described herein. Tiles can be constrained so that tiles grouped into the same tile group can form a rectangular region within the image. The concept of wrapping tile groups can be considered a modification of the rectangular shaped tile groups. Tiles can be grouped along Figure 6BThe vertical direction and / or along Figure 6C The horizontal direction is shown. Figure 6B In the example shown, the order of the chunks within the chunk group is as follows: chunk at index 19, chunk at index 20, chunk at index 21, chunk at index 22, chunk at index 1, chunk at index 2, chunk at index 3, and chunk at index 4. On the other hand, for Figure 6B In the example shown, the order of chunks within the chunk group is as follows: chunk at index 10, chunk at index 11, chunk at index 6, chunk at index 7, chunk at index 16, chunk at index 17, chunk at index 12, and chunk at index 14.

[0104] In a specific example, the variables numTileRowsInTileGroup, numTileColumnsInTileGroup, and NumTilesInTileGroup can be derived as follows:

[0105] deltaTileIdx=(bottom_right_tile_id <top_left_tile_id)?bottom_right_tile_id–top_left_tile_id+NumTilesInPic:((top_left_tile_id%(num_tile_columns_minus1+1))<(bottom_right_tile_id%(num_tile_columns_minus1+1)))?bottom_right_tile_id–top_left_tile_id+num_tile_columns_minus1:bottom_right_tile_id–top_left_tile_id

[0106] numTileRowsInTileGroup=(deltaTileIdx / (num_tile_columns_minus1+1))+1

[0107] numTileColumnsInTileGroup=(deltaTileIdx%(num_tile_columns_minus1+1))+1

[0108] NumTilesInTileGroup=numTileRowsInTileGroup*numTileColumnsInTileGroup

[0109] The variable TgTileIdx[i] may represent the tile index of the i-th tile in the tile group and may be derived as follows:

[0110]

[0111]

[0112] Through the above deduction, the definition of the syntax table of the block group data can be as follows.

[0113]

[0114] The arbitrary tile group concept allows flexibility in grouping two or more tiles into a group regardless of their location in the image. However, some systems may constrain the tiles in an arbitrary tile group to conform to the order of increasing tile index. To support this concept, the definition of the tile group indication can be as follows:

[0115]

[0116]

[0117] arbitrary_tile_group_flag may be set equal to 1 to indicate that each tile index of tiles in a tile group is explicitly assigned using the syntax element delta_tile_idx[i]. arbitrary_tile_group_flag may be set equal to 0 to indicate that the tile index of tiles in a tile group is identified by the syntax elements first_tile_id and last_tile_id. num_remaining_tiles_in_tile_group_minus1 may indicate the number of tiles in a tile group, excluding the first tile of the tile group. num_remaining_tiles_in_tile_group_minus1 may be Ceil(Log2(NumTilesInPic–1)) bits in length.

[0118] The variable NumTilesInTileGroup can be derived as follows:

[0119] NumTilesInTileGroup=num_remaining_tiles_in_tile_group_minus1+2

[0120] delta_tile_idx_minus1[i] may represent the difference between the tile index of the i-th tile and the tile index of the i–1-th tile in the tile group. The tile index may be derived from delta_tile_idx[i] and may be indicated in increasing order. The value of the tile index may not be equal to any other tile index in any other coded tile group of the same coded picture. The variable TgTileIdx[i] may be derived as follows:

[0121] TgTileIdx[i]=(i>0)? TgTileIdx[i–1]+delta_tile_idx_minus1[i]+1:first_tile_id

[0122] The above syntax and semantic description of the surrounding and arbitrary chunk groups may not include the concept of explicit chunk IDs, so the chunk ID and chunk index are the same.

[0123] This method of wrapping and arbitrarily shaped tile groups (or simply arbitrary tile groups) has at least the following problems. Arbitrary shaped tile groups allow flexible tile grouping. However, in practice, arbitrary tile grouping may be rarely used and only used in specific application areas, such as 360-degree video for spherical projection. Since this usage may be rare, it is inefficient to require that the syntax element arbitrary_tile_group_flag is always indicated when a tile group has multiple tiles. In addition, in order to support arbitrarily shaped tile groups, the tiles belonging to the tile group can be explicitly listed in the tile group header. Compared with indicating a rectangular tile group by indicating the first tile index or ID and the last tile index or ID, this naturally uses more bits. In order to improve the indication efficiency, the disclosed method only indicates the incremental block index between two consecutive tiles in the tile group. This may cause problems when using explicit tile ID indication, because deriving the tile index of the tile in the tile group may result in an incorrect tile index list.

[0124] Additionally, some surround block group indications may only support surround in the vertical direction or the horizontal direction, but not both. For example, some indication methods may not support Fig.6D When the block group is a surrounding block group in the vertical direction, the exemplary surrounding block group indication system allows the last block and the first block in the block group to be located in consecutive block rows. For example, in Figure 6B In , the first block is the block at index 19 and the last block is the block at index 16. Similarly, when the block group is a horizontally surrounding block group, the surrounding block group allows the last block and the first block in the block group to be located in consecutive block columns. For example, in Figure 6C, the first block is the block at index 10 and the last block is the block at index 15. Such a condition may not be needed because such groups of blocks are simple rectangular shapes and there is no other benefit in allowing such groupings of blocks as they may introduce unnecessary computational complexity to the decoder.

[0125] In order to solve the above problems, the present invention includes the following exemplary aspects. A flag is defined in a parameter set, which is directly or indirectly referenced by a tile group. The flag indicates whether a tile group of arbitrary shape is used. When the flag indicates that a tile group of arbitrary shape is used, a syntax element related to the tile group of arbitrary shape may be included in a tile group header and may (directly or indirectly) reference the parameter set. The flag may be indicated in a sequence parameter set, an image parameter set, or other types of parameter sets directly or indirectly referenced by a tile group, etc. The flag may be referred to as arbitrary_tile_group_present_flag.

[0126] In some aspects, when the chunk group is an arbitrary chunk group, the increment between the chunk IDs of two consecutive chunks in the chunk group may be indicated, rather than the increment between the chunk index values ​​of two consecutive chunks in the chunk group. In an alternative, when the chunk group is an arbitrary chunk group, the chunk ID may be indicated directly for each chunk in the chunk group, excluding the first chunk. This may be used as an alternative to indicating the increment between the chunk index values ​​of two consecutive chunks in the chunk group.

[0127] For wrapping block groups, wrapping in two directions (e.g., vertical and horizontal) may be supported, as shown below. Vertical and horizontal wrapping may be identified by the following conditions: the block index of the first block is greater than the block index of the last block, and the block column index of the first block is greater than the block column of the last block, assuming that the block column index is defined in increasing order from left to right. Alternatively, vertical and horizontal wrapping may be identified by the following conditions: the block row index of the first block is greater than the block index of the last block, and the block column index of the first block is greater than the block column of the last block, assuming that the block column index is defined in increasing order from left to right, and the block row index is defined in increasing order from top to bottom.

[0128] The list of segment indexes associated with segments that belong to a surrounding segment group in both directions is described below. When explicit segment ID indication is not enabled, any surrounding segment group that satisfies one of the following conditions may not be allowed. A surrounding segment group where the segment row index of the first segment is greater than the segment row index of the last segment, and the difference between these segment row indices is 1. A surrounding segment group where the segment column index of the first segment is greater than the segment column index of the last segment, and the difference between these segment column indices is 1.

[0129] Specific exemplary implementations of the disclosed mechanisms are described below. An exemplary picture parameter set RBSP syntax is as follows.

[0130]

[0131]

[0132] tile_id_len_minus1+1 may represent the number of bits used to represent the following syntax elements: the syntax element tile_id_val[i][j] (when present in the PPS), and the syntax elements first_tile_id and last_tile_id in the tile group header of the referenced PPS. The value of tile_id_len_minus1 may be in the range of Ceil(Log2(NumTilesInPic) to 15, inclusive. arbitrary_tile_group_present_flag may be set equal to 1 to indicate that the syntax element arbitrary_tile_group_flag is present in the tile group header of the referenced PPS. arbitrary_tile_group_present_flag may be set equal to 0 to indicate that the syntax element arbitrary_tile_group_flag is not present in the tile group header of the referenced PPS.

[0133] An example chunk group header syntax is as follows:

[0134]

[0135]

[0136] single_tile_in_tile_group_flag may be set equal to 1 to indicate that there is only one tile in a tile group. single_tile_in_tile_group_flag may be set equal to 0 to indicate that there are multiple tiles in a tile group. first_tile_id may represent the tile ID of the first tile of the tile group. The length of first_tile_id may be tile_id_len_minus1+1 bits. The value of first_tile_id may not be equal to the value of first_tile_id of any other coded tile group of the same coded image. When there are multiple tile groups in an image, the decoding order of the tile groups in the image may be an increasing value of first_tile_id. arbitrary_tile_group_flag may be set equal to 1 to indicate that the tiles in a tile group may be any set of two or more tiles of the image, and the tile ID of each tile is explicitly indicated in the tile group header. arbitrary_tile_group_flag may be set equal to 0 to indicate that the tiles in a tile group are identified by the syntax elements first_tile_id and last_tile_id. When not present, the value of arbitrary_tile_group_flag is inferred to be equal to 0. last_tile_id may represent the tile ID of the last tile of the tile group. The length of last_tile_id is tile_id_len_minus1+1 bits. If not present, the value of last_tile_id is inferred to be equal to first_tile_id.

[0137] When arbitrary_tile_group_flag is equal to 0, the variables numTileRowsInTileGroup, numTileColumnsInTileGroup, and NumTilesInTileGroup can be derived as follows:

[0138]

[0139]

[0140] When explicit_tile_id_flag is equal to 0, the following codestream consistency constraints may apply. If firstTileRowIdx is greater than lastTileRowIdx, then the difference between firstTileRowIdx and lastTileRowIdx shall be greater than 1. If firstTileColumnIdx is greater than lastTileColumnIdx, then the difference between firstTileColumnIdx and lastTileColumnIdx shall be greater than 1. When arbitrary_tile_group_flag is equal to 0, the variable TgTileIdx[i] represents the tile index of the i-th tile in the tile group and may be derived as follows:

[0141]

[0142] num_remaining_tiles_in_tile_group_minus1 may represent the number of tiles in a tile group, excluding the first tile of the tile group. The value of num_remaining_tiles_in_tile_group_minus1 may be in the range of 0 to NumTilesInPic–1, inclusive. When arbitrary_tile_group_flag is equal to 1, the variable NumTilesInTileGroup may be derived as follows:

[0143] NumTilesInTileGroup=num_remaining_tiles_in_tile_group_minus1+2

[0144] delta_tile_id_minus1[i] may represent the difference between the tile ID of the (i+1)th tile and the tile ID of the i-th tile in the tile group. The value of delta_tile_id_minus1[i] may be in the range of 0 to 2(tile_id_len_minus1+1)–1, inclusive. When arbitrary_tile_group_flag is equal to 1, for i in the range of 0 to NumTilesInTileGroup–1, inclusive, the variable TgTileIdx[i] may be derived as follows:

[0145] tgTileId[i]=i>0? tgTileId[i–1]+delta_tile_id_minus1[i–1]+1:first_tile_id

[0146] TgTileIdx[i]=TileIdToIdx[tgTileId[i]]

[0147] An example overall chunk group data syntax is as follows:

[0148]

[0149]

[0150] Figure 7 Schematic diagram of an exemplary video decoding device 700. As described herein, the video decoding device 700 is suitable for implementing the disclosed examples / embodiments. The video decoding device 700 includes a downstream port 720, an upstream port 750 and / or a transceiver unit (Tx / Rx) 710, and the transceiver unit (Tx / Rx) 710 includes a transmitter and / or a receiver for transmitting data upstream and / or downstream through a network. The video decoding device 700 also includes a processor 730, which includes a logic unit and / or a central processing unit (CPU) for processing data and a memory 732 for storing data. The video decoding device 700 may also include electrical, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to the upstream port 750 and / or the downstream port 720 for transmitting data through an electrical, optical or wireless communication network. The video decoding device 700 may also include an input and / or output (I / O) device 760 for communicating data with a user. The I / O device 760 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 760 may also include an input device, such as a keyboard, a mouse, a trackball, etc., and / or a corresponding interface for interacting with these output devices.

[0151] The processor 730 is implemented by hardware and software. The processor 730 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 730 communicates with the downstream port 720, Tx / Rx 710, upstream port 750, and memory 732. The processor 730 includes a decoding module 714. The decoding module 714 implements the disclosed embodiments described herein, such as methods 100, 800, 900, and 1000, which can use the code stream 500 and / or images divided into block groups 609, 619, 629, and / or 639. The decoding module 714 can also implement any other methods / mechanisms described herein. In addition, the decoding module 714 can implement the encoding and decoding system 200, the encoder 300, and / or the decoder 400. For example, when used as an encoder, the decoding module 714 can divide the image into a block group that surrounds the image in the horizontal and / or vertical direction. The decoding module 714 can also encode the flag in the parameter set to represent any block group and indicate the first block index and the last block index of the block group. When used as a decoder, the decoding module 714 can read the flag representing any block group and the first block index and the last block index of the block group. In the encoder or decoder, the decoding module 714 can determine the nature of the block group that surrounds the image, and adjust the incremental block index accordingly before using the incremental block index to determine the configuration of the block group. Therefore, the decoding module 714 improves the function of the video decoding device 700, and solves the specific problems in the field of video decoding by allowing multiple types of block groups to surround (therefore improving decoding efficiency) without negatively affecting the block group indication. In addition, the decoding module 714 can transform the video decoding device 700 to different states. Alternatively, the decoding module 714 may be implemented as instructions stored in the memory 732 and executed by the processor 730 (eg, a computer program product stored in a non-transitory medium).

[0152] The memory 732 includes one or more memory types, such as a disk, a tape drive, a solid-state drive, a read only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 732 can be used as an overflow data storage device to store programs when such programs are selected for execution, and to store instructions and data read during program execution.

[0153] Figure 8 Flow chart of an exemplary method 800 for encoding an image (e.g., images 601, 611, 621, and 631) having any partition groups (e.g., partition groups 609, 619, 629, and 639) into a bitstream (e.g., bitstream 500). Method 800 may be used by an encoder (e.g., codec system 200, encoder 300, and / or video decoding device 700) when performing method 100.

[0154] Method 800 may start when an encoder receives a video sequence including a plurality of images and determines to encode the video sequence into a bitstream based on user input, etc. Before encoding, the video sequence is segmented into images / frames for further segmentation. In step 801, the image is segmented into a plurality of blocks. The blocks may be further segmented into a plurality of CTUs, which may be further segmented into coding blocks for prediction-based compression. A group of blocks is also assigned to a block group. In a specific example, the block group is a rectangular shape that surrounds the image in the horizontal and vertical directions.

[0155] In step 803, the tile group is encoded into the bitstream. In addition, a flag is encoded into a parameter set in the bitstream. The flag indicates that the tile group is an arbitrary tile group. As a specific example, the flag is arbitrary_tile_group_present_flag. For example, the flag can be encoded into a PPS associated with the image.

[0156] In step 805, the first block index and the last block index can be encoded in the block group header. The first block index represents the first block in the first block group. In addition, the last block index represents the last block in the first block group. The first block index and the last block index can be encoded to support the number of blocks in the block group from the image segmentation at the decoder. In addition, the encoder can determine the number of blocks in the block group from the image segmentation according to the first block index and the last block index, as a part of the rate-distortion optimization process, while predicting the result at the decoder.

[0157] In step 807, the code stream is stored for transmission to the decoder. The code stream may be transmitted to the decoder upon request.

[0158] Fig. 9 Flow chart of an exemplary method 900 for decoding an image (e.g., images 601, 611, 621, and 631) having any partition groups (e.g., partition groups 609, 619, 629, and 639) from a code stream (e.g., code stream 500). Method 900 may be used by a decoder (e.g., codec system 200, decoder 400, and / or video decoding device 700) when performing method 100.

[0159] For example, after the method 800 ends, when the decoder starts receiving a code stream representing the encoded data of the video sequence, the method 900 may be started. In step 901, the code stream is received at the decoder. The code stream includes a block group, which includes a group of blocks segmented from an image.

[0160] In step 903, a flag is obtained from a parameter set in the code stream. The flag indicates that the tile group is an arbitrary tile group. As a specific example, the flag is arbitrary_tile_group_present_flag. For example, the flag can be obtained from the PPS associated with the image. Based on the flag, the decoder can select a predefined algorithm to determine the configuration of tiles in the tile group.

[0161] In step 905, the first block index and the last block index of the block group can be obtained from the block group header. The first block index indicates the first block in the block group. The last block index indicates the last block in the block group.

[0162] In step 907, the number of segments in the segment group can be determined based on the flag, the first segment index and the last segment index. As a specific example, the segment group can be a rectangular shape that surrounds the image in the horizontal and vertical directions. A predefined algorithm can be used to determine the number of segments in the segment group, regardless of the type of surround that the segment group uses.

[0163] In step 909, the first group of partitions may be decoded to reconstruct a portion of the image, which may then be included as part of a reconstructed video sequence. The resulting reconstructed video sequence may be forwarded to a display device for display to a user.

[0164] Fig.10 1 is a flow chart of an exemplary method 1000 for determining a partition group configuration from any partition group (e.g., partition groups 609, 619, 629, and / or 639) based on a first partition index and a last partition index. For example, the method 1000 may be performed at an encoder as part of a rate-distortion optimization process, or at a decoder in step 907 as part of the method 900. Therefore, the method 1000 may be used to support the methods 800 and 900. Therefore, when performing the method 100 for example on the bitstream 500, the video decoding device (e.g., the codec system 200, the encoder 300, the decoder 400, and / or the video encoding device 700) may use the method 1000.

[0165] When the video decoding device decides to determine the number of blocks in a block group segmented from an image according to a flag indicating that the block group is an arbitrary block group, method 1000 may start. In step 1001, the incremental block index is set to the difference between the last block index of the block group and the first block index of the block group.

[0166] In step 1003, the incremental block index is modified according to the block group surround. This can be done before determining the number of block rows in the block group and the number of block columns in the block group in steps 1005 and 1007. In the first example, when the last block index of the block group is less than the first block index of the block group (representing vertical surround), and when the first block column index is greater than the last block column index (representing horizontal surround), the incremental block index increases the number of blocks in the image and the number of block columns in the image. In the second example, when the last block index of the block group is less than the first block index of the block group (representing vertical surround), and when the first block column index is not greater than the last block column index (representing no horizontal surround), the incremental block index increases the number of blocks in the image. In the third example, when the last block index of the block group is not less than the first block index of the block group (representing no vertical surround), and when the first block column index is greater than the last block column index (representing horizontal surround), the incremental block index increases the number of block columns in the image.

[0167] In step 1005, the number of tile rows in the tile group is determined by dividing the incremental tile index by the number of tile columns in the image plus one. In step 1007, the number of tile columns in the tile group is determined as the incremental tile index modulo the number of tile columns in the second image plus one. In addition, the number of tiles in the tile group is determined by multiplying the number of tile rows in the tile group by the number of tile columns in the tile group.

[0168] Fig.11 1 is a schematic diagram of an exemplary system 1100 for decoding a video sequence of images (e.g., images 601, 611, 621, and 631) in a bitstream (e.g., bitstream 500). System 1100 may be implemented by an encoder and a decoder (e.g., encoding and decoding system 200, encoder 300, decoder 400, and / or video decoding device 700). In addition, system 1100 may be used to implement methods 100, 800, 900, and / or 1000.

[0169] The system 1100 includes a video encoder 1102. The video encoder 1102 includes a segmentation module 1101 for segmenting a first image into a plurality of partitions. The video encoder 1102 also includes an allocation module 1103 for allocating a group of partitions to a first partition group, wherein the first partition group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction. The video encoder 1102 also includes an encoding module 1105 for encoding the first partition group into a bitstream and encoding a flag into a parameter set in the bitstream to indicate that the first partition group is an arbitrary partition group. The video encoder 1102 also includes a storage module 1107 for storing the bitstream for sending to a decoder. The video encoder 1102 also includes a sending module 1109 for sending a bitstream to support determining the configuration of the partition group and decoding the partition in the decoder. The video encoder 1102 can also be used to perform any step in method 800 and / or 1000.

[0170] The system 1100 also includes a video decoder 1110. The video decoder 1110 includes a receiving module 1111 for receiving a code stream, the code stream including a first block group, the first block group including a group of blocks segmented from a first image. The video decoder 1110 also includes an acquisition module 1113 for acquiring a flag from a parameter set in the code stream, wherein the flag indicates that the first block group is an arbitrary block group. The video decoder 1110 also includes a determination module 1115 for determining the number of blocks in the first block group according to the flag, the first block index and the last block index, wherein the first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction. The video decoder 1110 also includes a decoding module 1117 for decoding the first block group to generate a reconstructed video sequence for display. The video decoder 1110 can also be used to perform any step in method 900 and / or 1000.

[0171] A first component is directly coupled to a second component when there is no intermediate component between the first component and the second component except a line, trace or other medium. A first component is indirectly coupled to a second component when there is an intermediate component between the first component and the second component except a line, trace or other medium. The term "coupled" and its synonyms include direct coupling and indirect coupling. Unless otherwise specified, the term "approximately" means a range of ±10% of the number that follows it.

[0172] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and some steps may be omitted or combined.

[0173] Although several embodiments have been provided in the present invention, it is understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit and scope of the present invention. The present examples are considered to be illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0174] In addition, the techniques, systems, subsystems and methods described and shown as discrete or separate in various embodiments may be combined or integrated with other systems, components, techniques or methods without departing from the scope of the present invention. Other examples of changes, substitutions and modifications may be determined by those skilled in the art, and changes, substitutions and modifications may be made without departing from the spirit and scope of the present invention.

Claims

1. A method implemented in an encoder, characterized in that The method comprises: The processor of the encoder divides the first image into a plurality of blocks; The processor assigns a group of the blocks to a first block group, wherein the first block group is a rectangular shape surrounding the first image in a horizontal direction and a vertical direction; The processor encodes the first block group into a bitstream; The processor encodes a flag into a parameter set in the code stream to indicate that the first block group is an arbitrary block group; Storing the bitstream in a memory of the encoder, wherein the bitstream is used to be sent to a decoder; The method further includes determining the number of blocks in the second group of blocks segmented from the second image by: Setting the incremental chunk index to the difference between the last chunk index of the second chunk group and the first chunk index of the second chunk group; determining the number of tile rows in the second tile group by dividing the incremental tile index by the number of tile columns in the second image plus one; The number of tile columns in the second tile group is determined as the incremental tile index modulo the number of tile columns in the second image plus one.

2. The method according to claim 1, characterized in that The flag is arbitrary_tile_group_present_flag.

3. The method according to claim 1 or 2, characterized in that: The flag is encoded into a picture parameter set (PPS) associated with the first picture.

4. The method according to claim 1 or 2, characterized in that: The method also includes encoding, by the processor, a first partition index and a last partition index into a partition group header, wherein the first partition index indicates a first partition in the first partition group, and the last partition index indicates a last partition in the first partition group.

5. The method according to claim 1 or 2, characterized in that: Before determining the number of block rows in the second block group and the number of block columns in the second block group, when the last block index of the second block group is less than the first block index of the second block group, and when the first block column index is greater than the last block column index, the incremental block index increases the number of blocks in the second image and the number of block columns in the second image.

6. The method according to claim 1 or 2, characterized in that: Before determining the number of block rows in the second block group and the number of block columns in the second block group, the incremental block index increases the number of blocks in the second image when the last block index of the second block group is less than the first block index of the second block group and when the first block column index is not greater than the last block column index.

7. The method according to claim 1 or 2, characterized in that: Before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is not less than the first block index of the second block group, and when the first block column index is greater than the last block column index, the incremental block index increases the number of the block columns in the second image.

8. A method implemented in a decoder, characterized in that: The method comprises: The processor of the decoder receives a code stream through a receiver, the code stream comprising a parameter set and a first block group, the first block group comprising a group of blocks segmented from a first image; The processor obtains a flag from the parameter set, wherein the flag indicates that the first block group is an arbitrary block group; The processor determines the number of blocks in the first block group according to the flag, the first block index and the last block index, wherein the first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction; The processor decodes the first group of blocks to generate a reconstructed video sequence for display; The method further includes determining the number of blocks in the second group of blocks segmented from the second image by: Setting the incremental chunk index to the difference between the last chunk index of the second chunk group and the first chunk index of the second chunk group; determining the number of tile rows in the second tile group by dividing the incremental tile index by the number of tile columns in the second image plus one; The number of tile columns in the second tile group is determined as the delta tile index modulo the number of tile columns in the second image plus one.

9. The method according to claim 8, characterized in that The flag is arbitrary_tile_group_present_flag.

10. The method according to claim 8 or 9, characterized in that: The flag is obtained from a picture parameter set (PPS) associated with the first picture.

11. The method according to claim 8 or 9, characterized in that: Also includes: The processor obtains the first chunk index and the last chunk index from a chunk group header, wherein the first chunk index indicates a first chunk in the first chunk group, and the last chunk index indicates a last chunk in the first chunk group.

12. The method according to claim 8 or 9, characterized in that: Before determining the number of block rows in the second block group and the number of block columns in the second block group, when the last block index of the second block group is less than the first block index of the second block group, and when the first block column index is greater than the last block column index, the incremental block index increases the number of blocks in the second image and the number of block columns in the second image.

13. The method according to claim 8 or 9, characterized in that: Before determining the number of block rows in the second block group and the number of block columns in the second block group, the incremental block index increases the number of blocks in the second image when the last block index of the second block group is less than the first block index and when the first block column index is not greater than the last block column index.

14. The method according to claim 8 or 9, characterized in that: Before determining the number of the block rows in the second block group and the number of the block columns in the second block group, when the last block index of the second block group is not less than the first block index and when the first block column index is greater than the last block column index, the incremental block index increases the number of the block columns in the second image.

15. A video decoding device, characterized in that: include: A processor, a receiver coupled to the processor, and a transmitter coupled to the processor, the processor, the receiver and the transmitter being configured to perform the method according to any one of claims 1 to 14.

16. A non-transitory computer readable medium, characterized in that A computer program product for use in a video decoding device, the computer program product comprising computer executable instructions stored in the non-transitory computer readable medium, and when a processor executes the computer executable instructions, the video decoding device performs the method according to any one of claims 1 to 14.

17. An encoder, characterized in that: The encoder is also used to perform the method according to any one of claims 1 to 7; the encoder comprises: A segmentation module, used for segmenting the first image into a plurality of blocks; an allocating module, configured to allocate a group of the blocks to a first block group, wherein the first block group is a rectangular shape surrounding the first image in a horizontal direction and a vertical direction; Encoding module for: Encoding the first block group into a bitstream; encoding a flag into a parameter set in the bitstream to indicate that the first block group is an arbitrary block group; A storage module, used for storing the code stream for sending to a decoder; A determination module is used to determine the number of blocks in a second block group segmented from a second image by the following steps: Setting the incremental chunk index to the difference between the last chunk index of the second chunk group and the first chunk index of the second chunk group; determining the number of tile rows in the second tile group by dividing the incremental tile index by the number of tile columns in the second image plus one; The number of tile columns in the second tile group is determined as the delta tile index modulo the number of tile columns in the second image plus one.

18. A decoder, characterized in that: The decoder is also used to perform the method according to any one of claims 8 to 14; the decoder comprises: A receiving module, configured to receive a code stream, wherein the code stream includes a first block group, and the first block group includes a group of blocks segmented from a first image; An acquisition module, configured to acquire a flag from a parameter set in the bitstream, wherein the flag indicates that the first block group is an arbitrary block group; a determination module, configured to determine the number of blocks in the first block group according to the flag, the first block index and the last block index, wherein the first block group is a rectangular shape surrounding the first image in the horizontal direction and the vertical direction; A decoding module, used for decoding the first block group to generate a reconstructed video sequence for display; The determination module is further configured to determine the number of blocks in the second block group segmented from the second image by the following steps: Setting the incremental chunk index to the difference between the last chunk index of the second chunk group and the first chunk index of the second chunk group; determining the number of tile rows in the second tile group by dividing the incremental tile index by the number of tile columns in the second image plus one; The number of tile columns in the second tile group is determined as the delta tile index modulo the number of tile columns in the second image plus one.

Images